Java implementation of the Internet Research Lab Web Crawler (IRLbot) as presented by Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov in their paper "IRLbot: Scaling to 6 Billion Pages and Beyond"
https://code.google.com/p/jkeylockmanager/ https://github.com/mojgh/JKeyLockManager
The following link shows a Stanford guest lecture of a Google employee who gives details on the evolution of the Google search and indexing system. Very interesting!
Video: Building Software Systems At Google and Lessons Learned