fix(user_tools): honor AWS profile S3 endpoints

[WilliamK112]

Description

Add a shared S3 endpoint resolver used by both the Python S3Fs handler and local Hadoop job argument construction. Existing explicit and environment behavior is preserved, with AWS profile configuration consulted only when no endpoint environment variable is set.

Endpoint precedence is:

1. explicit endpoint_override argument for S3Fs
2. AWS_ENDPOINT_URL_S3
3. AWS_ENDPOINT_URL
4. the active AWS profile from AWS_CONFIG_FILE or ~/.aws/config

Profile selection follows AWS_PROFILE, then AWS_DEFAULT_PROFILE, then default, and resolves exactly one active profile. If an explicitly selected profile has no endpoint, the resolver returns no override instead of falling through to another profile whose credentials are not active.

The selected profile supports profile-level endpoint_url, service-section S3 endpoints, and nested S3 endpoint configuration. Missing, malformed, or undecodable config files are ignored. AWS config is read as UTF-8 with interpolation disabled so percent-encoded endpoint URLs remain intact.

For local Java jobs, a resolved custom endpoint is also passed as the Hadoop S3A endpoint with path-style access enabled.

Fixes #2087.

Validation

Merged current dev (2073060) into the PR branch, then ran from user_tools:

• PYTHONPATH=src .venv/bin/python -m pytest tests/spark_rapids_tools_ut/test_csppath.py -q — 21 passed
• PYTHONPATH=src .venv/bin/python -m pytest tests/spark_rapids_tools_ut/test_csppath.py tests/spark_rapids_tools_ut/test_cluster.py -q — 28 passed
• Pylint on the parser and regression tests — 10.00/10
• Flake8 on the parser and regression tests
• py_compile on the parser
• git diff --check upstream/dev...HEAD

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable new defects or outstanding previous findings remain.

Summary

• Adds profile, services-section, and nested S3 endpoint resolution.
• Selects exactly one active profile without borrowing endpoints from fallback profiles.
• Safely ignores missing, malformed, and undecodable configuration while preserving percent-encoded URLs.
• Applies resolved endpoints to both S3Fs and Hadoop S3A configuration.
• Adds regression coverage for precedence, parsing, profile selection, and explicit overrides.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Resolve S3 endpoint] --> B{AWS_ENDPOINT_URL_S3 set?}
    B -->|Yes| C[Use S3 environment endpoint]
    B -->|No| D{AWS_ENDPOINT_URL set?}
    D -->|Yes| E[Use global environment endpoint]
    D -->|No| F[Select one active AWS profile]
    F --> G[Read AWS config without interpolation]
    G --> H{S3 endpoint configured?}
    H -->|Services or nested S3| I[Use service-specific endpoint]
    H -->|Profile-level only| J[Use profile endpoint]
    H -->|No| K[No endpoint override]
    C --> L[Python S3Fs and Hadoop S3A]
    E --> L
    I --> L
    J --> L

Loading

_{Reviews (3) · Last reviewed commit: "Merge remote-tracking branch 'upstream/d..."}

[WilliamK112]

I addressed the latest parser finding in 9eee17b, then merged current dev into the branch as requested for active PR updates (no rebase or force-push).

Fresh validation after the merge passes: 21 endpoint/path tests, 28 combined endpoint and cluster tests, Pylint 10.00/10, Flake8, py_compile, and git diff --check.

Could you please take another look when convenient?