Skip to content

Document PerfIO S3 client configuration - #15943

Open
gerashegalov wants to merge 1 commit into
NVIDIA:mainfrom
gerashegalov:fix/15919-s3-timeouts
Open

Document PerfIO S3 client configuration#15943
gerashegalov wants to merge 1 commit into
NVIDIA:mainfrom
gerashegalov:fix/15919-s3-timeouts

Conversation

@gerashegalov

Copy link
Copy Markdown
Collaborator

Related to #15919.

Description

The PerfIO S3 reader supports selecting Netty or CRT with the existing
spark.rapids.perfio.s3.httpClient setting, but that setting was omitted from the generated
advanced configuration documentation. The Iceberg S3 async tuning settings are specific to
PerfIO and should not be declared as core RapidsConf entries.

This change:

  • documents the existing PerfIO S3 HTTP client backend selector, including its supported values
    and default;
  • documents the three new Iceberg S3 request timeout settings and updates the existing settings
    with backend-neutral descriptions where appropriate; and
  • removes the PerfIO-only Iceberg S3 declarations from the core configuration registry while
    retaining their generated user documentation.

The existing configuration names and defaults are unchanged.

Validation:

  • ./build/buildall --clean --profile=350 --module=dist --parallel=64 --option='-DskipTests'
  • git diff --check

Performance testing is not required because this change only updates configuration metadata and
documentation; it does not alter request execution.

AI assistance: the implementation and validation were performed with Codex assistance and
reviewed by the author before publication.

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Document the existing PerfIO S3 HTTP client backend selector and the new request timeout settings. Remove PerfIO-only Iceberg S3 declarations from the core configuration registry while retaining their generated user documentation.

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
@gerashegalov gerashegalov added the documentation Improvements or additions to documentation label Sep 9, 2026
@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR aligns the advanced configuration reference with PerfIO’s S3 client configuration and removes duplicate PerfIO-only entries from the core RapidsConf registry.

  • Documents the Netty/CRT HTTP client selector and S3 timeout controls.
  • Makes existing S3 descriptions backend-neutral where appropriate.
  • Keeps configuration names and documented defaults unchanged while moving ownership outside the core registry.

Confidence Score: 5/5

The PR appears safe to merge; no concrete behavioral, documentation, security, or repository-rule issue remains.

The removed typed entries have no repository callers, and raw PerfIO settings continue to propagate through the prefix-filtered RAPIDS configuration map while configuration documentation is sourced through the generator’s local and classpath-provided metadata.

Important Files Changed

Filename Overview
docs/additional-functionality/advanced_configs.md Adds and updates advanced PerfIO S3 configuration documentation with backend selection, timeout defaults, and backend-neutral descriptions.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Removes unused PerfIO-specific Iceberg S3 declarations from the core configuration registry without changing raw configuration propagation.

Reviews (1): Last reviewed commit: "Document PerfIO S3 client configuration" | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants