Kernel argument parameters too large

[michel2323]

Backend: oneAPIBackend()
[ Info:  ... downloaded 17.0 B (100% complete, 213496.504 days)
[ Info:  ... downloaded 17.0 B (100% complete, 213496.504 days)
[ Info:  ... downloaded 17.0 B (100% complete, 213496.504 days)
[ Info:  ... downloaded 137.1 MiB (30% complete, 213496.504 days)
[ Info:  ... downloaded 182.8 MiB (40% complete, 213496.504 days)
[ Info:  ... downloaded 228.5 MiB (50% complete, 213496.504 days)
[ Info:  ... downloaded 274.2 MiB (60% complete, 213496.504 days)
[ Info:  ... downloaded 319.9 MiB (70% complete, 213496.504 days)
[ Info:  ... downloaded 365.5 MiB (80% complete, 213496.504 days)
[ Info:  ... downloaded 411.2 MiB (90% complete, 213496.504 days)
[ Info:  ... downloaded 456.1 MiB (100% complete, 213496.504 days)
┌ Error: Module compilation failed:
│
│ error: Total size of kernel arguments exceeds limit! Total arguments size: 2056, limit: 2048

Thanks to the help from ALCF (@kris-rowe), it could be resolved with:

export IGC_OverrideOCLMaxParamSize=4096

Here was the issue with CUDA.jl https://github.com/JuliaGPU/CUDA.jl/pull/2180/files#diff-ecbfaf5b99ab10dcafff5717c7cc5f856768e4313446fa3fb58b839a25b17cfc .

I guess we should find a way to poll for the hardware/compiler capabilities.

[maleadt]

OpenCL has CL_DEVICE_MAX_PARAMETER_SIZE; I don't see an equivalent in Level Zero.

[kris-rowe]

I think that ze_device_module_properties_t.maxArgumentsSize is the equivalent.

[maleadt]

Ah yes, my grep-fu was lacking this morning. Should be relatively easy to port that functionality, then.

[zippylab]

I was using export IGC_OverrideOCLMaxParamSize=4096 to work around a similar kernel-argument-size issue in AMReX, and learned that the PVC GPU has a hardware limit of 2048 bytes (kernel argument register block). The IGC_OverrideOCLMaxParamSize setting doesn't change the hardware limit, but only hides the compiler message about exceeding it. In the AMReX case, this led to silent wrong answers, as I think it would in any case where a single kernel argument exceeds 2048 bytes.