Prior to #1362 cudf spilling could be enabled either via client.run(cudf.set_option) or via the CUDF_SPILL env variable. #1362 adds a flag to enable cudf spilling that makes the process a lot simpler but doesn't work in cases where users are trying to enable spilling via the CUDF_SPILL environment variable rather than using the flag.
This results in the workers being initialized without cudf spill by default ignoring the env variable and leading to errors downstream.
Prior to #1362 cudf spilling could be enabled either via
client.run(cudf.set_option)or via theCUDF_SPILLenv variable. #1362 adds a flag to enable cudf spilling that makes the process a lot simpler but doesn't work in cases where users are trying to enable spilling via theCUDF_SPILLenvironment variable rather than using the flag.This results in the workers being initialized without cudf spill by default ignoring the env variable and leading to errors downstream.