nix: CUDA support

[attilaolah]

Summary

Describe the problem and the smallest change that solves it.

Validation

• make -C c check
• CUDA changes were tested with make -C c cuda-test (if applicable)
• Performance claims include hardware, commands, and repeatable measurements

Compatibility

• The default CPU build remains dependency-free
• No model files, generated binaries, or benchmark artifacts are included

[JustVugg]

Thanks for this, and sorry for the slow reply — the flake work is welcome and Nix users have asked for it.

Before merging I need a bit more from the PR itself, because right now I can't tell what it does or whether it works. The description is the unfilled template, all three validation boxes are unchecked, and the change is 119 lines across flake.nix/flake.lock with no commentary. I'd have to reverse-engineer the intent from the diff, and CUDA packaging is exactly the area where I can't guess safely — I have no NVIDIA hardware to check against.

Could you fill in:

1. What it does, in two or three sentences. What does the flake expose after this — a CUDA-enabled colibri? An extra output or devShell alongside the existing CPU one? Does the default output change for someone who runs nix build today with no GPU?
2. What you ran, on what. GPU, driver, CUDA version, and the exact commands — nix build .#<what> and then something proving the binary actually uses the GPU. The engine prints [CUDA] device 0: <name>, ... sm_XX on a successful init; that line in the output is the evidence I'm after.
3. Something that catches a regression. A flake check, or a CI job, or even just a documented command in the PR body that someone can rerun. As it stands, if this breaks in three months nothing tells us.

The compatibility question matters most to me: the CPU build must stay dependency-free by default. If nix build with no arguments still gives a plain CPU engine and CUDA is opt-in, say so explicitly and that concern is settled.

Not asking for a rewrite — the code may well be right. I just can't merge packaging for hardware I can't test on the strength of a diff alone. Fill those in and I'll look again properly.

[JustVugg]

Follow-up to my earlier note: this now also conflicts with dev (.gitignore and the tree have moved), and no CI has ever run on it — GitHub holds workflow runs from contributors whose first PR here hasn't merged, which is why the PR shows no checks at all. That part is on me and I'll approve the runs as soon as there's something to test.

So the state is: draft, conflicting, no CI, and the description is still the unfilled template with all three validation boxes unchecked. I'm not able to review CUDA packaging for hardware I don't own on the strength of a diff alone.

The three things I asked for still stand — what the flake exposes after this, what you ran and on which GPU/driver (with the [CUDA] device 0: ... banner line as evidence the binary really used the card), and something that catches a regression later. Add those plus a rebase and I'll look at it properly.

Genuinely happy to have Nix support; I just can't merge it blind.

[attilaolah]

Hey, sorry for leaving this in draft for so long. I didn't have much time to look at it these days.

I have a box with an RTX CUDA GPU I was testing this on, and I haven't made much progress since I haven't seen the speedups I was hoping for.

It builds with CUDA 13.2, 13.3 too but using that globally in NixOS right now is problematic due to a compiler bug that breaks the ONNX runtime (unrelated to this repo), that will be fixed in CUDA 13.4.

I saw no speedups vs. CPU then I realised my bottleneck was elsewhere. Then I found out I haven't downloaded the 8 bit quants for the speculative pass, I did so but then too I couldn't push it past 1 tok/s, so eventually I had no more time to debug this.

I'll get back to this in the next few days, ideally I'd also want to bake in metalSupport while there but that will be even more painful as my Mac has only 24G unified memory so I'll have so Stream a lot from storage.

But thanks for the reminder :) I'll get back to this soon and address your points too. Right now the branch is in a bit of an unfinished state, even though technically it still builds.

One problem I don't know how to solve yet is passing what GPU to compile against, Nix will build this in isolation so the build toolchain won't be able to detect a GPU the way it does with the CPU. Building in impure mode would be a workaround but a bit against the Nix philosophy IMO.

[tomberek]

builds and runs for CPU-only as expected