I have ran into major IP address blocking because of the number of requests. Recently, it happened with the fixed IP in my job and we received a question from our ISP. Not like most people fall into this but may harm crawling even in dynamic IP addresses environments. Maybe we should have an option to limit the number of requests per time to a specific website / domain / IP address.
I have ran into major IP address blocking because of the number of requests.
Recently, it happened with the fixed IP in my job and we received a question from our ISP.
Not like most people fall into this but may harm crawling even in dynamic IP addresses environments.
Maybe we should have an option to limit the number of requests per time to a specific website / domain / IP address.