[BUG] `SVR.fit` issues when prefetching enabled

[jcrist]

This was discovered in #7834. For reasons unknown to me, SVR.fit doesn't always work when rmm is configured to use a rmm.mr.PrefetchResourceAdaptor. This adaptor calls the internal mr, then calls cudaMemPrefetchAsync to schedule a prefetch of the pointer before returning.

I see two failure modes when enabling a PrefetchResourceAdaptor with SVR:

• PrefetchResourceAdaptor(ManagedMemoryResource()): the intercept is incorrect in cases where y is all the same value. Without prefetching you get the correct intercept, but with prefetching the intercept is still 0.
• PrefetchResourceAdaptor(CudaMemoryResource()): the fit call errors with a failure inside CUB.

Using a ManagedMemoryResource() or CudaMemoryResource() without prefetching works fine. Since the SVM memory management code is a bit hoky (and doesn't use rmm everywhere IIUC) I suspect this is an issue isolated to there. Raising to see if someone more versed in our C++ code can determine the issue.

The following simple pytest tests all cases, demonstrating both issues:

import numpy as np
import pytest
from cuml.svm import SVR
import rmm


@pytest.mark.parametrize("prefetch", [False, True])
@pytest.mark.parametrize("mode", ["cuda", "managed"])
def test_svr_constant(mode, prefetch):
    if mode == "cuda":
        mr = rmm.mr.CudaMemoryResource()
    elif mode == "managed":
        mr = rmm.mr.ManagedMemoryResource()

    if prefetch:
        mr = rmm.mr.PrefetchResourceAdaptor(mr)

    rmm.mr.set_current_device_resource(mr)

    X = np.eye(4)
    y = np.full(4, 1.234)
    m = SVR().fit(X, y)
    y_pred = m.predict(X)
    np.testing.assert_allclose(y, y_pred)

Output of `pytest test.py`

============================= test session starts ==============================
platform linux -- Python 3.13.12, pytest-8.4.2, pluggy-1.6.0
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/jcristharif/Code/cuml
configfile: pyproject.toml
plugins: xdist-3.8.0, cases-3.9.1, benchmark-5.2.3, cov-7.0.0, hypothesis-6.151.9
collected 4 items

test.py .F.F                                                             [100%]

=================================== FAILURES ===================================
_________________________ test_svr_constant[cuda-True] _________________________

mode = 'cuda', prefetch = True

    @pytest.mark.parametrize("prefetch", [False, True])
    @pytest.mark.parametrize("mode", ["cuda", "managed"])
    def test_svr_constant(mode, prefetch):
        if mode == "cuda":
            mr = rmm.mr.CudaMemoryResource()
        elif mode == "managed":
            mr = rmm.mr.ManagedMemoryResource()
    
        if prefetch:
            mr = rmm.mr.PrefetchResourceAdaptor(mr)
    
        rmm.mr.set_current_device_resource(mr)
    
        X = np.eye(4)
        y = np.full(4, 1.234)
>       m = SVR().fit(X, y)
            ^^^^^^^^^^^^^^^

test.py:22: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../miniforge3/envs/cuml-dev/lib/python3.13/site-packages/cuml/internals/outputs.py:417: in inner
    res = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
../../miniforge3/envs/cuml-dev/lib/python3.13/site-packages/cuml/svm/svr.py:194: in fit
    self._fit(X, y, sample_weight)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   RuntimeError: fill_n: failed inside CUB: cudaErrorInvalidDevice: invalid device ordinal

cuml/svm/svm_base.pyx:470: RuntimeError
_______________________ test_svr_constant[managed-True] ________________________

mode = 'managed', prefetch = True

    @pytest.mark.parametrize("prefetch", [False, True])
    @pytest.mark.parametrize("mode", ["cuda", "managed"])
    def test_svr_constant(mode, prefetch):
        if mode == "cuda":
            mr = rmm.mr.CudaMemoryResource()
        elif mode == "managed":
            mr = rmm.mr.ManagedMemoryResource()
    
        if prefetch:
            mr = rmm.mr.PrefetchResourceAdaptor(mr)
    
        rmm.mr.set_current_device_resource(mr)
    
        X = np.eye(4)
        y = np.full(4, 1.234)
        m = SVR().fit(X, y)
        y_pred = m.predict(X)
>       np.testing.assert_allclose(y, y_pred)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       Mismatched elements: 4 / 4 (100%)
E       Max absolute difference among violations: 1.234
E       Max relative difference among violations: inf
E        ACTUAL: array([1.234, 1.234, 1.234, 1.234])
E        DESIRED: array([0., 0., 0., 0.])

test.py:24: AssertionError
=========================== short test summary info ============================
FAILED test.py::test_svr_constant[cuda-True] - RuntimeError: fill_n: failed i...
FAILED test.py::test_svr_constant[managed-True] - AssertionError: 
========================= 2 failed, 2 passed in 1.88s ==========================

[viclafargue]

CUB's cudaErrorInvalidDevice error is caused by earlier attempt by RMM to prefetch non-managed memory leaving the CUDA context with a "sticky" error. rapidsai/rmm#2267 (or an alternative PR) should solve the issue. The assertion error in test_svr_constant[managed-True] is directly caused by the earlier CUDA context error. I would not be able to explain what is happening in detail at the moment, but I can confirm that all tests are passing when compiling with the fix.

[jcrist]

Thanks Victor. If I understand correctly from the linked discussion, PrefetchResourceAdaptor(CudaMemoryResource()) should be invalid (this wasn't clear to me from the docs). Perhaps rmm should error in that case or at least document it?

I'm confused about the other issue though (running with prefetched managed memory). I originally saw the issue without wrapping a CudaMemoryResource, and indeed can reproduce it just fine with only managed memory. Given that, I don't think

The assertion error in test_svr_constant[managed-True] is directly caused by the earlier CUDA context error.

is true?

Try running the following example - on my machine it errors just fine in a fresh process:

import rmm
import numpy as np
from cuml import SVR

mr = rmm.mr.PrefetchResourceAdaptor(rmm.mr.ManagedMemoryResource())

rmm.mr.set_current_device_resource(mr)

X = np.eye(4)
y = np.full(4, 1.234)
m = SVR().fit(X, y)
y_pred = m.predict(X)
np.testing.assert_allclose(y, y_pred)

Output:

$ python bug.py
Traceback (most recent call last):
  File "/home/jcristharif/Code/cuml/bug.py", line 13, in <module>
    np.testing.assert_allclose(y, y_pred)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/jcristharif/miniforge3/envs/cuml-dev/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 1715, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^
  File "/home/jcristharif/miniforge3/envs/cuml-dev/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 921, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 4 / 4 (100%)
Max absolute difference among violations: 1.234
Max relative difference among violations: inf
 ACTUAL: array([1.234, 1.234, 1.234, 1.234])
 DESIRED: array([0., 0., 0., 0.])

To be clear - this is the real bug I need resolved. Testing PrefetchResourceAdaptor(CudaMemoryResource()) was only found while trying to debug and trying a cartesian product of memory resources, it's not an actual use case we need to resolve.

[viclafargue]

Thanks for pushing back. The actual use case (PrefetchResourceAdaptor(ManagedMemoryResource)) indeed has an issue that is reproducible on its own and not caused by CUDA error of earlier test What made me say that is that building and installing the librmm fix solves all issues.

After looking a bit deeper at the issue it looks like even managed memory prefetching can fail with a cudaErrorInvalidValue. This happens when cudaMemPrefetchAsync is called with ptr=nullptr or size=0. RMM containers should be protected against this case. However, the SVM codepath had a potential empty allocation in the case in which it cannot find any support vector (edge case).

It looks like SVM was failing silently and was returning wrong predictions. #7859 should fix this issue by preventing empty allocation allowing the PrefetchResourceAdaptor(ManagedMemoryResource) use case to be merged (as the RMM fix may need more time to be discussed/reviewed).