Prebuilt Binaries

[Mythra]

👋🏻 Just stumbled across this project, and I'm interested in integrating this compiler into my CMake toolchains that I provide for others. I notice the README says

"It's not designed to be used as a standalone tool, but support for this may be added in the future."

Although I do see a PR was implemented for building + running an elf. However, it's currently only being distributed + built on linux. Would this repository entertain PRs for other platform support (e.g. building on macOS / windows)? Would this PR mind integrating CI steps to build + release those binaries (or me building + hosting them)? I'd like to avoid a fork, but also just don't want to start sending PRs if there's no interest here.

[Exzap]

We are pretty busy with other projects so this isn't really actively maintained right now. I wanted to revisit it eventually because Mesa made some pretty large changes to their R600 backend. It would streamline some things on our side (the way it works right now is quite hacky since we had to patch a lot of hardcoded constants in Mesa code directly). But it would probably also require a pretty large rewrite and since I lack the time this won't happen any time soon anyway.

With that said I think submitting PRs is fine. We'd appreciate Windows support. But if it's more convenient for you to maintain your own fork thats also alright with us.

[Mythra]

Thank You! I appreciate the info. I think what I'll do is I'll send some PRs for any windows support (or if it turns out to be needed Mac support), but I'll probably keep the prebuilt artifacts out of tree/PR for now. That way I can still distribute to people without any burden on y'all for merging PRs, and when this project does have more time for contribution, I can contribute and unify then.

[Mythra]

Do we have some example shader files that are available somewhere? I did check some:

• https://github.com/decaf-emu/decaf-emu/blob/master/tools/latte-assembler/resources/
• https://github.com/devkitPro/wut/tree/master/samples/cmake/content

However, these error out while compiling, using my new windows binary, and main:

$ ./build-host/cafecompiler/glslcompiler.elf -ps /tmp/test.psh -vs /tmp/test.vsh
ctx->Const.PackedDriverUniformStorage: 0
ctx->Const.NativeIntegers: 1
Compiling pixel shader: /tmp/test.psh
Shader failed to compile. Log:
0:1(3): error: syntax error, unexpected invalid token, expecting end of file

Shader /tmp/test.psh failed to compile:
0:1(3): error: syntax error, unexpected invalid token, expecting end of fil

At first I thought maybe my change was broken. main also appeared to be broken, so I thought "hey maybe main is broken", but I checked out a release and the same error occurred:

$ curl -sSL -o /tmp/g.elf "https://github.com/Exzap/CafeGLSL/releases/download/v0.2.0/glslcompiler.elf"
$ chmod +x /tmp/g.elf
$ /tmp/g.elf -ps /tmp/test.psh -vs /tmp/test.vsh
ctx->Const.PackedDriverUniformStorage: 0
ctx->Const.NativeIntegers: 1
Compiling pixel shader: /tmp/test.psh
Shader failed to compile. Log:
0:1(3): error: syntax error, unexpected invalid token, expecting end of file

Shader /tmp/test.psh failed to compile:
0:1(3): error: syntax error, unexpected invalid token, expecting end of file

Perhaps something is wrong with the release too, but am asking for validation before continuing.

[Exzap]

These work and were compiled with CafeGLSL: https://github.com/Crementif/UntitledSandGame/tree/main/source/assets/shaders

[Crementif]

That, plus we do have some embedded in the tests part of the source code: https://github.com/Exzap/CafeGLSL/blob/main/cafecompiler/tests.cpp#L72

[Mythra]

Thanks for those (and thanks @Crementif , i was aware but wanted some extra's just for more testing). So far, I've got something building with MSVC for amd64 + arm64 (in fact there are 0 warnings generated!). The current binary is segfault'ing on tests, which is why I wanted to make sure to get extra test materials.

The same build seems to build + run tests fine on linux/mac still even with all the changes. So there's something messy somewhere that will probably take me a couple days to work through. Specifically we end up with an executable that does the following before crashing:

> .\build-host\cafecompiler\glslcompiler.exe -t -v
Initialize compiler...

ctx->Const.PackedDriverUniformStorage: 0
ctx->Const.NativeIntegers: 1
Running compiler tests...

[DBG] [gl_nir_link_uniforms] Assing non-block uniform location 4
[DBG] [gl_nir_link_uniforms] Assing non-block uniform location 5
[DBG] [nir nir_lower_uniforms_to_ubo] Called
[DBG] [nir ubo lower_instr] Called
[DBG] [nir ubo lower_instr] Called
[DBG] [nir ubo lower_instr] Called
[DBG] [nir ubo lower_instr] Called
[DBG] [nir ubo lower_instr] Called
[DBG] [nir ubo lower_instr] Called
[DBG] [nir nir_lower_uniforms_to_ubo] progress loop var->data.binding 12 driver_location 140
[DBG] [nir nir_lower_uniforms_to_ubo] progress loop var->data.binding 12 driver_location 4294967295
[DBG] [nir nir_lower_uniforms_to_ubo] progress loop var->data.binding 10 driver_location 4294967295
[DBG] [nir nir_lower_uniforms_to_ubo] progress loop var->data.binding 10 driver_location 138
[DBG] [nir lower_deref] var->data.binding 2 var->data.explicit_binding 1
[DBG] [nir lower_deref] var->data.binding 2 var->data.explicit_binding 1
[DBG] [nir lower_deref] var->data.binding 4 var->data.explicit_binding 1
[DBG] [nir lower_deref] var->data.binding 4 var->data.explicit_binding 1
[DBG] [r600_get_varying_semantic] sid=0 (shifted: 9)
[DBG] [ShaderIO::set_sid] sid=9
[DBG] [r600_get_varying_semantic] sid=5 (shifted: 14)
[DBG] [ShaderIO::set_sid] sid=14
[DBG] [TexInstr::emit_lowered_tex] samplerId 2 textureId 2
[DBG] [TexInstr::emit_lowered_tex] samplerId 4 textureId 4

---

Dug into it a bit... Seems the crash comes out of sb_bitset::set_chk from the r600 driver, where an unsigned gets set to unsigned max:

inline bool sb_bitset::set_chk(unsigned id, bool bit) {
	assert(id < bit_size);
	unsigned w = id / bt_bits;
	unsigned b = id % bt_bits;
	fprintf(stderr, "(%u, %u, %u)\n", id, bit, w);
	basetype d = data[w];
	basetype dn = (d & ~(1 << b)) | (bit << b);
	bool r = (d != dn);
	data[w] = r ? dn : data[w];
	return r;
}

Prints: (4294967295, 0, 134217727) on windows, and (33, 0, 1) on linux (this is hit during the second and final "liveness" pass during shader linking). So that is definetly the faulting point. Classic Array OOB. I won't have time to check how the value id that gets passed gets constructed here yet, but i'm guessing it all comes down to some undefined behavior with wrapping and unsigned integers somewhere.

---

Unfortunately did some more debugging and this appears to be a UAF :(

[Mythra]

FWIW just to keep this issue up to date, although windows is now building (and seemingly compiling correctly see #7 ), macOS while it builds, does seem to segfault on certain compilations, so i'll be looking into fixing that.

[Mythra]

And: #8 should fix the last macOS failures. I've got to get some sleep, and got a lot to do tomorrow, but I think we should be fully good now. Once I get to actually writing the CMake, and integration into my toolchain I'll file issues if I see any, but I do thinkw e're in a good state now.

[Mythra]

Should be my second to last update (sorry about all the updates on this "maintenance mode" project!).

https://codeberg.org/rem-verse/temporary-cafe-glsl-builds now exists, and points to the links of prebuilt artifacts, CI, etc. I'm currently only building for Linux/macOS/Windows on amd64/arm64 however there's issue forms incase folks want other commits, or other OS's/arch's being built for.

I have only one more PR to file before I should be done for this issue. It will add a basic CMake pkg-config file, and renames the host binary only to a unique name (for finding installed binaries on the host system easily), so users writing projects in CMake (one of the supported options for DevkitPro!), can just easily pull in the executable, and run it. It is working for me on my toolchain alongside other devkitPro tools. I'm currently waiting on some reviews of the CMake from multiple CMake folks to make sure it's quality since I don't fully trust myself reviewing that.

For those who want to know how to "use" the compiler from CMake (who either isn't using my toolchain), you can see how i've exposed it to my toolchain below:

• https://codeberg.org/rem-verse/sprig/src/branch/trunk/toolchain/cmake/Modules/Compiler/DevkitPro-GNU.cmake#L137-L244 / https://codeberg.org/rem-verse/sprig/src/branch/trunk/toolchain/cmake/Modules/FindCafeGLSL.cmake: CMake macro that uses a pre-installed binary if possible (With the FindCafeGLSL.cmake), or will fetch the prebuilt artifact if it exists, or print an error.
• https://codeberg.org/rem-verse/sprig/src/branch/trunk/toolchain/cmake/Modules/Platform/DevkitPro.cmake#L41-L117: An "add_shaders" target that mirrors something like "add_executable", for compiling shaders.