RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation

[david-gasinski]

Hi @MasoudJTehrani,

First of all, thank you for the great work!

I'm currently investigating running multiple AD agents in 1 CARLA instance on a HPC environment. Whenever I run another agent (tfv3_tf for example) alongside carl_roach, my program crashes and prints:

RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation

I've tried various combinations of agents and the issue only occurs when carl_roach is present. I've boiled the issue down to pcla_agents/carl/eval_agent.py, where the CUDA kernel is set to deterministic.

# Make network deterministic.
torch.manual_seed(0)
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

You can disable the runtime error using torch.use_deterministic_algorithms(True, warn_only=True). However, this impacts the performance of the agents as not all use a deterministic kernel. Is there a way to set the kernel to only be deterministic for agents that need it? I'm planning to investigate this issue myself, potentially using a context manager approach to toggle the flag during specific agent steps. Would this be viable?

[MasoudJTehrani]

Hi David,
Thank you for posting this useful question.

Your idea of a context manager can work. But be careful of multithreading.

I also think there might be other ways of solving this problem. Try running the code for carl_roach in another Python file, both connecting to the same CARLA server. This way, you also need to use Python's subprocess or multiprocessing modules.

Let us know how you solved the problem