Building Arras on Windows

[BrianHanke]

Howdy all, I'm giving myself a fresh start on MoonRay for Windows. I recently built it on Rocky 9 in WSL and was surprised and pleased by how well MR works in that environment. So, I'm going to try the Hydra -> Arras -> MR on Linux route.

I've got the Arras dependencies installed via vcpkg. Configuring and generating the VS 2022 solution with CMake works fine. Before I continue, I've got a few general questions:

1. When working towards the above Hydra-centric workflow, is it ok to skip the arras_render Qt stuff?
2. So far it looks like the main problem is Linux-only headers, types and functions: pid_t, unistd.h, etc. Are there any resources for quickly addressing these issues on Windows? I'm sure it's possible to reinvent the wheel and fix each one from scratch, but I'm hoping there are existing tools or knowledge bases for these tasks.
3. Lastly, CMake errored out on ArrasCore::execute in two CMakeLists.txt files a few levels deep in the distributed folder. I just commented them out for now and CMake was able to finish, but I imagine this will have to be addressed at some point. Any idea why execute was a problem but not any of the others (arras4_log, network, etc)?

Many thanks!

[BrianHanke]

Regarding point 3, I see that ArrasCore::execute is Linux-only, so I guess that answers that question. Is it ok to leave it out entirely when porting to Windows?

[jlanz]

Hi Brian,
Thanks for your continued interest in MoonRay. I'll try to answer a few of your questions.

1. yes, the arras_render tool is used only for interactive rendering outside a Hydra workflow and can be skipped if not needed.

2. not that I am aware of. We are looking to replace all platform specific headers and source with cross platform solutions. Unfortunately, this will likely need to be done piece by piece.

3. if you can share more details on the problems you encountered we may be able to point you in the right direction. I don't think leaving it out will work if the intent is to include arras in a fully native windows workflow. Hybrid workflows where an arras client is running natively on Windows and the arras server is running on a different non-Windows host (or within WSL on the same host) should be possible.

[BrianHanke]

Thanks for the reply, Jon. I've made a fair bit of progress over here, cutting the number of build errors from over 200 to around 100 now.

I'm shooting for the "build Arras on Windows and eventually communicate with MoonRay in WSL" approach since that seems the easiest way to go.

Here's a quick list of some changes/observations/questions from my recent work:

• I get a CMake generation error on ArrasCore::execute in CMakeLists.txt in arras4_node/lib/session and arras4_node/cmd/node. I note that it's referenced under if (IsUnixPlatform) in CMakeLists.txt in arras4_core/arras4_client/lib/client/api, so that led me to think it's intended for Linux only. I commented it out just to get the generation done and start working on other things. I don't remember the exact error, but I can check later.
• I replaced sleep() in seconds with Windows Sleep() in milliseconds.
• I imported a Windows version of gettimeofday() from PostgreSQL which apparently is a drop-in replacement.
• I fixed a Boost path join error.
• I swapped out localtime_r for localtime_s, which required reversing the parameters but hopefully is the same otherwise.

I think that about covers it for now! Of course some, or all, of these changes may not actually work, but I'm hoping to just get the thing to build and then deal with other problems later.

Let me know what you think and if you have any suggestions! I'm determined to succeed and I feel confident that it's possible.

[BrianHanke]

Update: I've now got 16 out of 21 Arras modules building on Windows. Of the remaining 5, one just has .dll export problems, very easy to fix. The other 4 will need some actual code updates/rewrites. We'll see how that goes, but I'm very happy with the progress so far!

[BrianHanke]

Further update: 17 out of 21 Arras modules now build on Windows. The vast majority of the problems were due to .dll function exports, but that's all fixed now. I've also made a few code changes here and there as mentioned above (Sleep(), gettimeofday(), etc).

Of the 4 modules remaining, arras4_node has 24 errors, arras4_router 78, node_router 77 and node_session 3. All of these errors are due to Linux commands that don't work on Windows, such as kill() and poll(). 78 sounds like a lot, but it's mostly Visual Studio objecting to the same thing in several different ways. All in all I'd say it's just a few dozen lines total.

The task now is to find Windows replacements for these commands. I've had some luck with solutions that people have posted on GitHub and elsewhere, so hopefully that will continue.

[BrianHanke]

@jlanz I've now successfully compiled (most) of Arras on Windows! I have 18 libraries built plus arras4_router.exe.

I ran into trouble with arras4_node. As mentioned above, the CMake generate step fails as a result of ArrasCore::execute in two CMakeLists.txt files. I found a few lines in another CMakeLists.txt implying this module is intended for Linux only. That led me to comment out ArrasCore::execute and finish the generation step.

That allowed me to get as far as I have. However, arras4_node seems to require ProcessManager, which in turn requires the execute module. I'm not sure I'm understanding this correctly, it's a complicated business and I'm trying to piece together an understanding as best I can.

From the readme, it looks like arras4_node is needed only on the render node side of things. Ultimately I want to send jobs from Windows and render on Linux only, so I'm thinking arras4_node on Windows is irrelevant in my case. Am I on the right track with this? I'd be happy to leave Arras mostly complete (missing arras4_node only) and move on to other parts of the project.

[jlanz]

Correct, arras4_node need not be ported assuming you are planning on rendering on the linux side.

[BrianHanke]

@jlanz Following up on my above questions, I'm working on mcrt_dataio right now. I'm following the project order listed here: #106

However, the mcrt_dataio CMakeLists.txt shows McrtDenoise and SceneRdl2 as dependencies even though they're not in the first 4-item list.

Is this an oversight in that project order or should those two dependencies be removed when building on Windows? Thank you!

[jlanz]

Right, it does look like that first list of packages that would need to be ported to Windows is missing scene_rdl2 and mcrt_denoise.

[BrianHanke]

@jlanz Looking ahead to the next two in the list, Hydra and mcrt_computation, I see that they additionally list Moonray and Moonshine as dependencies. Can you please confirm if the original project order is accurate or if there are some dependencies missing from the list? Right now it looks to me like I'd have to add Moonray, Moonshine, McrtDenoise and SceneRdl2 at the very least.

[jlanz]

I think the best place to see the build dependencies is here:
https://docs.openmoonray.org/getting-started/installation/building-moonray/repo_deps/

It should be up to date, but if you notice anything off let us know.

[BrianHanke]

I think the best place to see the build dependencies is here: https://docs.openmoonray.org/getting-started/installation/building-moonray/repo_deps/

It should be up to date, but if you notice anything off let us know.

Thanks for the replies! That list gives a more complete picture for sure. I guess I'll dive into scene_rdl2 next. Already one issue is that the Visual Studio generator doesn't support ISPC, so I guess I'll have to use Ninja instead.

[BrianHanke]

@jlanz I've hit a wall with scene_rdl2. I got it to successfully generate with Ninja without any unusual warnings. However, upon trying to build the project Ninja isn't happy:

ninja: error: 'lib/common/math/ispc/Transcendental_ispc_stubs.h', needed by 'lib/common/math/common_math.lib', missing and no known rule to make it

I've tried lots of searches and LLMs, but building with Ninja and ISPC on Windows is such an esoteric thing that there is almost no related info that I can find. Do you have any suggestions? Thanks!

[BrianHanke]

@jlanz I'm still at an impasse with the ISPC stuff. I've been in touch with some ISPC experts and it looks like there's no chance in the near future of CMake being able to generate a VS solution when ISPC is involved. So, I'm stuck with Ninja.

Do you have any suggestions how to proceed? What build system do you use internally for Windows? I would have thought that since Ninja generated everything successfully it would build, but no such luck. Like I said, this is esoteric stuff and I can't find anything helpful online. I've made such good progress up until this point that I hate to give up now.

Thanks!

[jlanz]

Hi @BrianHanke - we don't build for Windows internally and I am not sure I can be of much help. I do know that in some early Windows tests one of our developers mentioned that he used ninja, but unfortunately I don't know much more.