Infinite path loop

[ethicalhack3r]

Hi,

It seems that when $_SERVER['REQUEST_URI'] or similar is used AND the web server is configured to return custom error pages (including 200 statuses), Spidr ends up in an infinite loop.

In this particular case the problem URL is in a POST form action element, but I don't think it matters where the URL appears.

Eventually ends up with pages like so:

http://www.example.com/dir/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/js/somefile.js

I'm not sure how this could be solved, the depth option may help cut down on the false positive URLs but wouldn't solve the problem.

Thanks,
Ryan

[postmodern]

If I could get a link to the page that triggers this that would be awesome; HTTP dump/pcap is also acceptable :) I want to rule out any server-side bugs, where these links are being generated by merely appending "/js" to the Request URI.

If it's sensitive, you can send me a PGPed email or priv me on IRC.

[postmodern]

I recently fixed a path expansion bug in URI.expand_path. Wondering if a04f368 also fixes this?