Problem
llama.cpp builds CUDA kernels for every quantization type × attention pattern combination, even when we're only using one or two quant types. This results in 100+ .cu.o compilations and 20-30 min cold builds on EC2.
Proposed Fix
Add a QUANT_TYPES variable to ec2-tool-calls-gen.sh (and related gen scripts) that limits template instantiation at build time:
# e.g. only build kernels needed for iq4_nl
QUANT_TYPES="iq4_nl"
Translate this into the appropriate CMake flags before the build step — likely via -DGGML_CUDA_FA_ALL_QUANTS=OFF and targeted -DCMAKE_CUDA_FLAGS overrides, or by patching the template instance CMakeLists before compilation.
Combined with the GPU arch already auto-detected (sm${GPU_ARCH}), this should cut cold build times significantly for single-quant jobs.
Notes
- llama.cpp has some coarse flags (
GGML_CUDA_FA_ALL_QUANTS etc.) but nothing fine-grained out of the box
- The S3 build cache key would need to incorporate quant types:
sm${GPU_ARCH}_cuda${CUDA_VER}_${QUANT_TYPES}
- Apply to both
ec2-tool-calls-gen.sh and ec2-gpu-expert-gen.sh
Problem
llama.cpp builds CUDA kernels for every quantization type × attention pattern combination, even when we're only using one or two quant types. This results in 100+
.cu.ocompilations and 20-30 min cold builds on EC2.Proposed Fix
Add a
QUANT_TYPESvariable toec2-tool-calls-gen.sh(and related gen scripts) that limits template instantiation at build time:Translate this into the appropriate CMake flags before the build step — likely via
-DGGML_CUDA_FA_ALL_QUANTS=OFFand targeted-DCMAKE_CUDA_FLAGSoverrides, or by patching the template instance CMakeLists before compilation.Combined with the GPU arch already auto-detected (
sm${GPU_ARCH}), this should cut cold build times significantly for single-quant jobs.Notes
GGML_CUDA_FA_ALL_QUANTSetc.) but nothing fine-grained out of the boxsm${GPU_ARCH}_cuda${CUDA_VER}_${QUANT_TYPES}ec2-tool-calls-gen.shandec2-gpu-expert-gen.sh