Fess Crawler is a crawler library for crawling a web site and a file system.
Web/FileSystem Crawler Library