Unreliable downloads of past versions

[h-vetinari]

CI on the maintenance branches is currently failing with a high incidence rate (~80% of jobs!), see 20.x, 19.x, 18.x; I've seen similar behaviour in the past few days already, though with more manageable failure rates.

Details

Downloading https://www.apache.org/dyn/closer.lua/arrow/arrow-20.0.0/apache-arrow-20.0.0.tar.gz?action=download
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='archive.apache.org', port=443): Read timed out. (read timeout=9.15)")': /dist/arrow/arrow-20.0.0/apache-arrow-20.0.0.tar.gz
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='archive.apache.org', port=443): Read timed out. (read timeout=9.15)")': /dist/arrow/arrow-20.0.0/apache-arrow-20.0.0.tar.gz
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='archive.apache.org', port=443): Read timed out. (read timeout=9.15)")': /dist/arrow/arrow-20.0.0/apache-arrow-20.0.0.tar.gz
WARNING: Error: HTTP 000 CONNECTION FAILED for url <[https://www.apache.org/dyn/closer.lua/arrow/arrow-20.0.0/apache-arrow-20.0.0.tar.gz?action=download>](https://www.apache.org/dyn/closer.lua/arrow/arrow-20.0.0/apache-arrow-20.0.0.tar.gz?action=download%3E)
Elapsed: -

This is one reason among several why I wanted to use the more reliable tarballs from the github tags (c.f. discussion in apache/arrow#32572), but upstream arrow maintainers have repeatedly recommended to use the https://www.apache.org/dyn/closer.lua?action=[...] URL, leading ultimately to a switch in dd20501

CC @raulcd @kou @assignUser

[assignUser]

This is a known problem with archive.a.o as it has pretty strict usage limits (strict when looking at the gha ip ranges, not normal use). Things are happening ASF internally to improve that but it's going to take a while.

For releases > 17.0.0 we do publish the same official tarball to Github releases, which can be used: https://github.com/apache/arrow/releases/tag/apache-arrow-21.0.0
For >20 we have also added a lot of the convenience binaries like wheels.

[h-vetinari]

For releases > 17.0.0 we do publish the same official tarball to Github releases

That's great news, thanks! I'll gradually switch the branches over to that then!