Skip to content

ec2 build: narrow CUDA kernel compilation by quant type and GPU arch #2

Description

@kmbandy

Problem

llama.cpp builds CUDA kernels for every quantization type × attention pattern combination, even when we're only using one or two quant types. This results in 100+ .cu.o compilations and 20-30 min cold builds on EC2.

Proposed Fix

Add a QUANT_TYPES variable to ec2-tool-calls-gen.sh (and related gen scripts) that limits template instantiation at build time:

# e.g. only build kernels needed for iq4_nl
QUANT_TYPES="iq4_nl"

Translate this into the appropriate CMake flags before the build step — likely via -DGGML_CUDA_FA_ALL_QUANTS=OFF and targeted -DCMAKE_CUDA_FLAGS overrides, or by patching the template instance CMakeLists before compilation.

Combined with the GPU arch already auto-detected (sm${GPU_ARCH}), this should cut cold build times significantly for single-quant jobs.

Notes

  • llama.cpp has some coarse flags (GGML_CUDA_FA_ALL_QUANTS etc.) but nothing fine-grained out of the box
  • The S3 build cache key would need to incorporate quant types: sm${GPU_ARCH}_cuda${CUDA_VER}_${QUANT_TYPES}
  • Apply to both ec2-tool-calls-gen.sh and ec2-gpu-expert-gen.sh

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions