Support Maven mirror configuration for tools downloads

[parthosa]

Fixes #2098

This PR lets RAPIDS tools use a configured Maven mirror for direct artifact and metadata downloads. Maven Central remains the default when no environment variables are set.

What Changed

• Added RAPIDS_TOOLS_MAVEN_BASE_URL support in Scala WebCrawlerUtil.
• Added RAPIDS_TOOLS_MAVEN_USERNAME and RAPIDS_TOOLS_MAVEN_PASSWORD for Basic Auth on Maven metadata and artifact page requests.
• Added the same Maven URL override and auth support in user-tools download paths.
• Updated AutoTuner expected output tests so plugin URLs follow the configured Maven base URL.
• Kept OSS behavior unchanged: without env vars, tools still use Maven Central.

Why

Some tools CI paths call Maven URLs directly instead of going through Maven settings. This includes the AutoTuner latest-plugin check:

AutoTuner -> WebCrawlerUtil.getLatestPluginRelease -> maven-metadata.xml

Using env vars lets CI point those direct downloads at an internal mirror and avoid Maven Central throttling.

Env Vars

RAPIDS_TOOLS_MAVEN_BASE_URL=<maven-mirror-url>
RAPIDS_TOOLS_MAVEN_USERNAME=<maven-user>
RAPIDS_TOOLS_MAVEN_PASSWORD=<maven-password>

Testing

• ToolUtilsSuite with Maven mirror env set: passed.
• Full core Maven test run with Maven mirror env set: passed.
• User-tools unit tests for Maven URL rewrite and auth headers: passed.
• Verified profiling CI failures are now only expected-result diffs where old golden output points to Maven Central and current output points to the mirror.

[amahussein]

For the scala webrawler thing: Do we actually need this feature? It felt like we needed in old days when the tools would point out to an outdated RAPIDS version. However, the RAPIDS added the buildEventInfo that dumps the information about the rapids jar used on the job. Given that RAPIDS is released bi-monthly we can get approximation that the RAPIDS is out-dated without doing any webcrawls.
IF we want to be realy precise on minor releases, then we can probe the minor version incrementally before we hit a miss.

In that case we won't need any webcrawling anymore.

On Python/build side:

• For someone configuring his mvn settings to use internal archive, mvn command should automatically work fine and fetch the jars from the internal repo.
• Github would still use the public repo which is fine as long as no one is using it for testing builds.

[parthosa]

For the scala webrawler thing: Do we actually need this feature? It felt like we needed in old days when the tools would point out to an outdated RAPIDS version. However, the RAPIDS added the buildEventInfo that dumps the information about the rapids jar used on the job. Given that RAPIDS is released bi-monthly we can get approximation that the RAPIDS is out-dated without doing any webcrawls. IF we want to be realy precise on minor releases, then we can probe the minor version incrementally before we hit a miss.

In that case we won't need any webcrawling anymore.

In the AutoTuner path, I agree that removing the web lookup entirely would be cleaner long term. We would have to figure out the semantics for getting an approximation that the plugin is out-dated.

On Python/build side:

• For someone configuring his mvn settings to use internal archive, mvn command should automatically work fine and fetch the jars from the internal repo.
• Github would still use the public repo which is fine as long as no one is using it for testing builds.

For the Python/fat-wheel side, Maven settings do not cover the failing path. The CSP dependencies are read from JSON configs and downloaded through Python.

For this MR, I wanted to have a less invasive fix by keeping the existing behavior, but let CI point the lookup to the internal Maven mirror in both cases.

[greptile-apps]

Confidence Score: 3/5

The PR should not merge until authenticated Maven requests are prevented from transmitting credentials over plaintext HTTP.

Findings

1.   Credentials Allow Plaintext Transport ▶

Summary

• Rewrites existing Maven Central resource URLs to the configured mirror.
• Adds authenticated metadata, page, and artifact requests.
• Updates AutoTuner URL expectations and adds mirror/authentication tests.
• Authentication currently permits plaintext HTTP endpoints, exposing configured credentials in that configuration.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  E[Maven environment variables] --> R[Resolve repository base]
  R --> S[Scala WebCrawlerUtil]
  R --> P[Python Utilities]
  S --> M1[Metadata and artifact-page requests]
  P --> M2[Metadata requests]
  P --> D[DownloadTask artifact requests]
  A[Username and password] --> H[Basic Authorization header]
  H --> M1
  H --> M2
  H --> D

Loading

[greptile-apps]

Missing connection and read timeouts on Maven URL connection

URLConnection.openConnection() uses the JVM default timeouts (effectively infinite). If the configured mirror is slow or unresponsive, connection.getInputStream can block indefinitely, stalling whichever CI job or tool invocation triggered getLatestMvnReleaseForNVPackage. The old code via XML.load(url) had the same problem, but now that an explicit openConnection() is exposed, it's straightforward to set timeouts. Consider calling connection.setConnectTimeout(...) and connection.setReadTimeout(...) before getInputStream.

[greptile-apps]

Credentials Allow Plaintext Transport

If RAPIDS_TOOLS_MAVEN_BASE_URL uses http://, this code accepts the URL and the reachable metadata request attaches the configured Basic-auth username and password. The Scala implementation behaves the same way, and its new test explicitly combines an HTTP endpoint with credentials. A network observer can therefore recover the credentials. Reject authenticated non-HTTPS endpoints or only attach credentials to HTTPS URLs.

How this was verified: Both URL builders accept an arbitrary configured scheme, and their reachable metadata request paths attach the Basic Authorization header before opening that URL.