blender: fix segfault when using both HIP and OSL

[frozolotl]

This PR fixes an issue where enabling both ROCm and OpenShadingLanguage in Blender causes Blender to crash when performing a Cycles render.

Blender dlopens HIP, which is dynamically linked to rocm-comgr, which in turn links ROCm's libLLVM.
But then, Blender dynamically links OpenShadingLanguage, which links libLLVM 19.
And so, when HIP is invoked, it transitively calls ROCm's libLLVM, which then—due to symbol conflicts—calls functions in libLLVM 19.
One of those functions then dereferences a null pointer, causing a segfault in the process.

Switching out dlopen with dlmopen separates the namespaces of the two libLLVMs, fixing the issue.

If anyone wants to find a better solution, I'll mention that simply using the same version of LLVM for both ROCm and OpenShadingLanguage causes different issues because then libLLVM is initialized twice.

The bug was also mentioned in #502842.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

[drupol]

Great ! Would it be a good idea to upstream this patch ?

[frozolotl]

It appears as though this bug swatted itself? The latest master appears to work correctly now.

I think I'm more confused now than I was when submitting this PR, but hey, it's a good kind of confusion.

https://projects.blender.org/blender/blender/pulls/156456#issuecomment-1888351

EDIT: even #502842 works now?! I'm certain I had that issue happen on those exact commits. @amarshall too. It's weird.
The segfault does still happen on 5fab99f though. So it's not like I just can't reproduce it anymore.

@amarshall can you check whether your results are the same as mine?