Yesterday tests/test_proxify_host_file.py::test_worker_force_spill_to_disk started to present intermittent failures in CI. This is the first time it occurred, with two jobs failing, after rerunning the jobs twice (see the attempts in the same link), at least one more of the jobs still failed. Today's nightlies all passed, but then another failure occurred in a PR run. This issue is happening in 25.10, so if it's a legitimate issue it will likely affect release.
I suspect this is due to some third-party package upgrade but I don't have anything more specific about where this could be coming from. I have compiled a diff of packages from a failed job from yesterday where this was first observed and the equivalent job from the day before that below.
Package diff
--- test-967-success.txt 2025-09-26 17:32:07
+++ test-968-fail.txt 2025-09-26 17:31:18
@@ -15,8 +15,8 @@
aws-c-s3 0.8.6 h4e5ac4b_5 conda-forge
aws-c-sdkutils 0.2.4 h92c474e_1 conda-forge
aws-checksums 0.2.7 h92c474e_2 conda-forge
-aws-crt-cpp 0.34.3 h60c762c_1 conda-forge
-aws-sdk-cpp 1.11.606 h2d2fbd4_3 conda-forge
+aws-crt-cpp 0.34.4 h60c762c_0 conda-forge
+aws-sdk-cpp 1.11.606 h32384e2_4 conda-forge
azure-core-cpp 1.16.0 h3a458e0_1 conda-forge
azure-identity-cpp 1.12.0 ha729027_0 conda-forge
azure-storage-blobs-cpp 12.14.0 hb1c9500_1 conda-forge
@@ -28,7 +28,7 @@
c-ares 1.34.5 hb9d3cd8_0 conda-forge
ca-certificates 2025.8.3 hbd8a1cb_0 conda-forge
cachetools 6.2.0 pyhd8ed1ab_0 conda-forge
-cffi 1.17.1 py311h5b438cf_1 conda-forge
+cffi 2.0.0 py311h5b438cf_0 conda-forge
click 8.3.0 pyh707e725_0 conda-forge
cloudpickle 3.1.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_1 conda-forge
@@ -55,16 +55,16 @@
cuda-pathfinder 1.2.3 pyhcf101f3_0 conda-forge
cuda-python 12.9.2 pyhb6702c7_2 conda-forge
cuda-version 12.2 he2b69de_3 conda-forge
-cudf 25.10.00a429 cuda12_py311_250924_ffa7ea2d rapidsai-nightly
+cudf 25.10.00a434 cuda12_py311_250925_2de829c9 rapidsai-nightly
cupy 13.6.0 py311h72da3fd_2 conda-forge
cupy-core 13.6.0 py311he30c881_2 conda-forge
cytoolz 1.0.1 py311h9ecbd09_0 conda-forge
dask 2025.9.1 pyhcf101f3_0 conda-forge
dask-core 2025.9.1 pyhcf101f3_0 conda-forge
-dask-cuda 25.10.00a27 py311_250924_3145a9f5 file:///tmp/tmp.u0wxo1sam7
-dask-cudf 25.10.00a429 cuda12_py311_250924_ffa7ea2d rapidsai-nightly
+dask-cuda 25.10.00a27 py311_250925_3145a9f5 file:///tmp/tmp.UFGUg5yfMI
+dask-cudf 25.10.00a434 cuda12_py311_250925_2de829c9 rapidsai-nightly
distributed 2025.9.1 pyhcf101f3_0 conda-forge
-distributed-ucxx 0.46.00a47 py_250923_5df29cce_h9c9281c rapidsai-nightly
+distributed-ucxx 0.46.00a48 py_250925_e09a7d5f_ha650c70 rapidsai-nightly
dlpack 0.8 h59595ed_3 conda-forge
exceptiongroup 1.3.0 pyhd8ed1ab_0 conda-forge
fastrlock 0.8.3 py311hc665b79_2 conda-forge
@@ -80,16 +80,16 @@
jinja2 3.1.6 pyhd8ed1ab_0 conda-forge
keyutils 1.6.3 hb9d3cd8_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
-kvikio 25.10.00a49 cuda12_py311_250924_1b70488a rapidsai-nightly
+kvikio 25.10.00a49 cuda12_py311_250925_1b70488a rapidsai-nightly
lcms2 2.17 h717163a_0 conda-forge
ld_impl_linux-64 2.44 h1423503_1 conda-forge
lerc 4.0.0 h0aef613_1 conda-forge
libabseil 20250512.1 cxx17_hba17884_0 conda-forge
-libarrow 21.0.0 h7838d49_5_cuda conda-forge
-libarrow-acero 21.0.0 hb826db4_5_cuda conda-forge
-libarrow-compute 21.0.0 h58682fd_5_cuda conda-forge
-libarrow-dataset 21.0.0 hb826db4_5_cuda conda-forge
-libarrow-substrait 21.0.0 h9d9f3f8_5_cuda conda-forge
+libarrow 21.0.0 h73424eb_6_cpu conda-forge
+libarrow-acero 21.0.0 h635bf11_6_cpu conda-forge
+libarrow-compute 21.0.0 h8c2c5c3_6_cpu conda-forge
+libarrow-dataset 21.0.0 h635bf11_6_cpu conda-forge
+libarrow-substrait 21.0.0 h3f74fd7_6_cpu conda-forge
libblas 3.9.0 36_h4a7cf45_openblas conda-forge
libbrotlicommon 1.1.0 hb03c661_4 conda-forge
libbrotlidec 1.1.0 hb03c661_4 conda-forge
@@ -98,7 +98,7 @@
libcblas 3.9.0 36_h0358290_openblas conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcublas 12.2.5.6 hd3aeb46_0 conda-forge
-libcudf 25.10.00a429 cuda12_250924_ffa7ea2d rapidsai-nightly
+libcudf 25.10.00a434 cuda12_250925_2de829c9 rapidsai-nightly
libcufft 11.0.8.103 hd3aeb46_0 conda-forge
libcufile 1.7.2.10 hd3aeb46_0 conda-forge
libcufile-dev 1.7.2.10 hd3aeb46_0 conda-forge
@@ -126,7 +126,7 @@
libgrpc 1.73.1 h1e535eb_0 conda-forge
libiconv 1.18 h3b78370_2 conda-forge
libjpeg-turbo 3.1.0 hb9d3cd8_0 conda-forge
-libkvikio 25.10.00a49 cuda12_9_250924_1b70488a rapidsai-nightly
+libkvikio 25.10.00a49 cuda12_9_250925_1b70488a rapidsai-nightly
liblapack 3.9.0 36_h47877c9_openblas conda-forge
liblzma 5.8.1 hb9d3cd8_2 conda-forge
libnghttp2 1.67.0 had1ee68_0 conda-forge
@@ -139,11 +139,11 @@
libopenblas 0.3.30 pthreads_h94d23a6_2 conda-forge
libopentelemetry-cpp 1.21.0 hb9b0907_1 conda-forge
libopentelemetry-cpp-headers 1.21.0 ha770c72_1 conda-forge
-libparquet 21.0.0 h006c7c7_5_cuda conda-forge
+libparquet 21.0.0 h790f06f_6_cpu conda-forge
libpng 1.6.50 h421ea60_1 conda-forge
libprotobuf 6.31.1 h9ef548d_1 conda-forge
libre2-11 2025.08.12 h7b12aa8_1 conda-forge
-librmm 25.10.00a38 cuda12_250924_278d5596 rapidsai-nightly
+librmm 25.10.00a38 cuda12_250925_278d5596 rapidsai-nightly
libsqlite 3.50.4 h0c1763c_0 conda-forge
libssh2 1.11.1 hcf80075_0 conda-forge
libstdcxx 15.1.0 h8f9b012_5 conda-forge
@@ -151,7 +151,7 @@
libsystemd0 257.9 h996ca69_0 conda-forge
libthrift 0.22.0 h454ac66_1 conda-forge
libtiff 4.7.1 h8261f1e_0 conda-forge
-libucxx 0.46.00a47 cuda12_250923_5df29cce rapidsai-nightly
+libucxx 0.46.00a48 cuda12_250925_e09a7d5f rapidsai-nightly
libudev1 257.9 h085a93f_0 conda-forge
libutf8proc 2.11.0 hb04c3b8_0 conda-forge
libuuid 2.41.2 he9a06e4_0 conda-forge
@@ -190,10 +190,10 @@
psutil 7.1.0 py311h49ec1c0_0 conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
pyarrow 21.0.0 py311h38be061_0 conda-forge
-pyarrow-core 21.0.0 py311hbce1db7_0_cuda conda-forge
+pyarrow-core 21.0.0 py311h342b5a4_0_cpu conda-forge
pycparser 2.22 pyh29332c3_1 conda-forge
pygments 2.19.2 pyhd8ed1ab_0 conda-forge
-pylibcudf 25.10.00a429 cuda12_py311_250924_ffa7ea2d rapidsai-nightly
+pylibcudf 25.10.00a434 cuda12_py311_250925_2de829c9 rapidsai-nightly
pysocks 1.7.1 pyha55dd90_7 conda-forge
pytest 8.4.2 pyhd8ed1ab_0 conda-forge
pytest-cov 7.0.0 pyhcf101f3_1 conda-forge
@@ -210,7 +210,7 @@
re2 2025.08.12 h5301d42_1 conda-forge
readline 8.2 h8c095d6_2 conda-forge
rich 14.1.0 pyhe01879c_0 conda-forge
-rmm 25.10.00a38 cuda12_py311_250923_278d5596 rapidsai-nightly
+rmm 25.10.00a38 cuda12_py311_250925_278d5596 rapidsai-nightly
s2n 1.5.26 h5ac9029_0 conda-forge
setuptools 80.9.0 pyhff2d567_0 conda-forge
six 1.17.0 pyhe01879c_1 conda-forge
@@ -224,7 +224,7 @@
typing_extensions 4.15.0 pyhcf101f3_0 conda-forge
tzdata 2025b h78e105d_0 conda-forge
ucx 1.19.0 hc93acc0_4 conda-forge
-ucxx 0.46.00a47 cuda12_py311_250923_5df29cce rapidsai-nightly
+ucxx 0.46.00a48 cuda12_py311_250925_e09a7d5f rapidsai-nightly
urllib3 2.5.0 pyhd8ed1ab_0 conda-forge
wheel 0.45.1 pyhd8ed1ab_1 conda-forge
xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge
@TomAugspurger I'll likely not have time to look into this today so it might be a problem that will need to be investigated next week as it will likely continue causing nightly errors.
Yesterday
tests/test_proxify_host_file.py::test_worker_force_spill_to_diskstarted to present intermittent failures in CI. This is the first time it occurred, with two jobs failing, after rerunning the jobs twice (see the attempts in the same link), at least one more of the jobs still failed. Today's nightlies all passed, but then another failure occurred in a PR run. This issue is happening in 25.10, so if it's a legitimate issue it will likely affect release.I suspect this is due to some third-party package upgrade but I don't have anything more specific about where this could be coming from. I have compiled a diff of packages from a failed job from yesterday where this was first observed and the equivalent job from the day before that below.
Package diff
@TomAugspurger I'll likely not have time to look into this today so it might be a problem that will need to be investigated next week as it will likely continue causing nightly errors.