diff --git a/docs/additional-functionality/advanced_configs.md b/docs/additional-functionality/advanced_configs.md index 5b5b60be78a..5f469ca87f9 100644 --- a/docs/additional-functionality/advanced_configs.md +++ b/docs/additional-functionality/advanced_configs.md @@ -39,10 +39,13 @@ Name | Description | Default Value | Applicable at spark.rapids.flameGraph.pathPrefix|Enables collecting flame graph (with async profiler) and specifies a file prefix to use when writing the JFR file by async-profiler. The async-profiler will write a flame graph file for each stage. It is strongly recommended to set 'spark.scheduler.mode' to 'FIFO' so that there is a clean boundary between stages, and then we can better understand each stage.|None|Runtime spark.rapids.flameGraph.stageEpochInterval|Interval in seconds to determine the current stage epoch based on running task counts. The profiler will check which stage has the most running tasks and profile that stage during each epoch. This allows profiling when multiple stages run concurrently even if FIFO scheduling is already chosen.|5|Runtime spark.rapids.gpu.resourceName|The name of the Spark resource that represents a GPU that you want the plugin to use if using custom resources with Spark.|gpu|Startup -spark.rapids.iceberg.s3.async.connection-max-idle-time-ms|Connection-max-idle-time (ms) for the AwsCrtAsyncHttpClient used by the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.|300000|Startup -spark.rapids.iceberg.s3.async.max-concurrency|Max concurrent connections for the AwsCrtAsyncHttpClient used by the cuDF plugin Iceberg S3 byte-range reader. Used only when the Iceberg FileIO property `s3.crt.max-concurrency` is not set.|200|Startup -spark.rapids.iceberg.s3.async.tcp-keepalive-interval-ms|TCP keep-alive probe interval (ms) for the AwsCrtAsyncHttpClient used by the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.|60000|Startup -spark.rapids.iceberg.s3.async.tcp-keepalive-timeout-ms|TCP keep-alive probe timeout (ms) for the AwsCrtAsyncHttpClient used by the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.|30000|Startup +spark.rapids.iceberg.s3.async.api-call-attempt-timeout-ms|Maximum time (ms) for one attempt of an Iceberg S3 byte-range request, including the HTTP request and response body.|30000|Startup +spark.rapids.iceberg.s3.async.api-call-timeout-ms|Maximum total time (ms) for an Iceberg S3 byte-range request, including retries.|120000|Startup +spark.rapids.iceberg.s3.async.connection-max-idle-time-ms|Connection-max-idle-time (ms) for the async HTTP client used by the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.|300000|Startup +spark.rapids.iceberg.s3.async.max-concurrency|Max concurrent connections for the async HTTP client used by the cuDF plugin Iceberg S3 byte-range reader. Used only when the Iceberg FileIO property `s3.crt.max-concurrency` is not set.|200|Startup +spark.rapids.iceberg.s3.async.socket-timeout-ms|Maximum time (ms) that an Iceberg S3 response may stop making progress. Applied as the Netty read timeout or the CRT connection-health timeout. The minimum is 2000 ms because that is the minimum supported by CRT.|30000|Startup +spark.rapids.iceberg.s3.async.tcp-keepalive-interval-ms|TCP keep-alive probe interval (ms) for the CRT async HTTP client used by the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.|60000|Startup +spark.rapids.iceberg.s3.async.tcp-keepalive-timeout-ms|TCP keep-alive probe timeout (ms) for the CRT async HTTP client used by the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.|30000|Startup spark.rapids.memory.gpu.allocFraction|The fraction of available (free) GPU memory that should be allocated for pooled memory. This must be less than or equal to the maximum limit configured via spark.rapids.memory.gpu.maxAllocFraction, and greater than or equal to the minimum limit configured via spark.rapids.memory.gpu.minAllocFraction.|1.0|Startup spark.rapids.memory.gpu.debug|Provides a log of GPU memory allocations and frees. If set to STDOUT or STDERR the logging will go there. Setting it to NONE disables logging. All other values are reserved for possible future expansion and in the mean time will disable logging.|NONE|Startup spark.rapids.memory.gpu.oomDumpDir|The path to a local directory where a heap dump will be created if the GPU encounters an unrecoverable out-of-memory (OOM) error. The filename will be of the form: "gpu-oom--.hprof" where is the process ID, and the dumpId is a sequence number to disambiguate multiple heap dumps per process lifecycle|None|Startup @@ -53,6 +56,7 @@ Name | Description | Default Value | Applicable at spark.rapids.memory.pinnedPool.parallelInit.threads|Number of CPU threads used to initialize the pinned pool's backing memory, capped at the number of executor cores. Set to 'all' to use the number of executor cores. A value of 1 initializes the backing memory using cudaHostAlloc. Values greater than 1 instead pre-touch pages concurrently before pinning for faster initialization. This does not affect subsequent suballocator behavior. Note: on multi-NUMA systems, multithreaded initialization can scatter pages across nodes if you do not constrain placement in advance. Pages cannot be migrated once pinned.|all|Startup spark.rapids.perfio.gcs.enabled|Controls the Google Cloud Storage reader for improved performance in certain queries. When true, enables it. When false, disables it unconditionally. When unset (default), enables it on Google Cloud Dataproc when compatible connector classes are available; outside Dataproc it remains disabled. Explicit enablement fails at startup if the required classes are unavailable.|None|Startup spark.rapids.perfio.s3.enabled|Controls the AWS S3 reader for improved performance in certain queries. When true, enables it and throws at startup if no compatible HTTP client is on the classpath. When false, disables it unconditionally. When unset (default), enables it opportunistically if a compatible HTTP client is found, otherwise falls back to S3A with a warning. The presence of AWS SDK packages for Netty and/or CRT HTTP clients on the classpath is required. You can use Spark submit option `--packages software.amazon.awssdk:s3:2.22.12,software.amazon.awssdk:aws-crt-client:2.22.12` to achieve this. See https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html#crt-based-s3-client-depend|None|Startup +spark.rapids.perfio.s3.httpClient|Selects the async HTTP client backend for the PerfIO S3 reader. Supported values: NETTY, CRT|NETTY|Startup spark.rapids.python.concurrentPythonWorkers|Set the number of Python worker processes that can execute concurrently per GPU. Python worker processes may temporarily block when the number of concurrent Python worker processes started by the same executor exceeds this amount. Allowing too many concurrent tasks on the same GPU may lead to GPU out of memory errors. >0 means enabled, while <=0 means unlimited|0|Runtime spark.rapids.python.memory.gpu.allocFraction|The fraction of total GPU memory that should be initially allocated for pooled memory for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.allocFraction)), since the executor will share the GPU with its owning Python workers. Half of the rest will be used if not specified|None|Runtime spark.rapids.python.memory.gpu.maxAllocFraction|The fraction of total GPU memory that limits the maximum size of the RMM pool for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.maxAllocFraction)), since the executor will share the GPU with its owning Python workers. when setting to 0 it means no limit.|0.0|Runtime diff --git a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala index 301f7379f5d..3e0c89fa93d 100644 --- a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala +++ b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala @@ -1878,39 +1878,6 @@ val GPU_COREDUMP_PIPE_PATTERN = conf("spark.rapids.gpu.coreDump.pipePattern") .booleanConf .createWithDefault(true) - val ICEBERG_S3_ASYNC_MAX_CONCURRENCY = - conf("spark.rapids.iceberg.s3.async.max-concurrency") - .doc("Max concurrent connections for the AwsCrtAsyncHttpClient used by the " + - "cuDF plugin Iceberg S3 byte-range reader. Used only when the Iceberg " + - "FileIO property `s3.crt.max-concurrency` is not set.") - .startupOnly() - .integerConf - .createWithDefault(200) - - val ICEBERG_S3_ASYNC_CONNECTION_MAX_IDLE_MS = - conf("spark.rapids.iceberg.s3.async.connection-max-idle-time-ms") - .doc("Connection-max-idle-time (ms) for the AwsCrtAsyncHttpClient used by the " + - "cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.") - .startupOnly() - .longConf - .createWithDefault(5L * 60 * 1000) - - val ICEBERG_S3_ASYNC_TCP_KEEPALIVE_INTERVAL_MS = - conf("spark.rapids.iceberg.s3.async.tcp-keepalive-interval-ms") - .doc("TCP keep-alive probe interval (ms) for the AwsCrtAsyncHttpClient used by " + - "the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.") - .startupOnly() - .longConf - .createWithDefault(60L * 1000) - - val ICEBERG_S3_ASYNC_TCP_KEEPALIVE_TIMEOUT_MS = - conf("spark.rapids.iceberg.s3.async.tcp-keepalive-timeout-ms") - .doc("TCP keep-alive probe timeout (ms) for the AwsCrtAsyncHttpClient used by " + - "the cuDF plugin Iceberg S3 byte-range reader. No equivalent Iceberg property.") - .startupOnly() - .longConf - .createWithDefault(30L * 1000) - val ENABLE_HIVE_TEXT: ConfEntryWithDefault[Boolean] = conf("spark.rapids.sql.format.hive.text.enabled") .doc("When set to false disables Hive text table acceleration. " +