feat: multi-method speculation cascade (--methods); CI benches MTP vs… #4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: arm64-cloud-benchmark | |
| # Free Arm64-cloud benchmark: runs on GitHub-hosted Arm runners (free for public repos, | |
| # no credit card). Builds llama.cpp with Arm KleidiAI kernels, fetches Gemma 4 + its MTP | |
| # head, and benchmarks self-draft. Doubles as a reproducible production CI workflow. | |
| on: | |
| workflow_dispatch: {} # run manually from the Actions tab | |
| push: | |
| branches: [ main ] | |
| paths-ignore: # don't re-run the expensive bench on docs-only changes | |
| - '**.md' | |
| - 'docs/**' | |
| - 'results/**' | |
| - 'patches/**' | |
| jobs: | |
| bench: | |
| runs-on: ubuntu-24.04-arm # free Arm64 (aarch64) runner for public repositories | |
| timeout-minutes: 120 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Show Arm host | |
| run: | | |
| uname -m # aarch64 | |
| nproc; free -h; df -h / | |
| - name: Install deps | |
| run: sudo apt-get update -y && sudo apt-get install -y build-essential cmake git python3 libcurl4-openssl-dev | |
| - name: Build llama.cpp with KleidiAI (Arm Neoverse i8mm/SVE2) | |
| run: | | |
| git clone --depth 1 https://github.com/ggml-org/llama.cpp.git | |
| cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release -DGGML_CPU_KLEIDIAI=ON -DGGML_NATIVE=ON | |
| cmake --build llama.cpp/build -j"$(nproc)" --target llama-server | |
| - name: Fetch Gemma 4 12B (Q4_0, KleidiAI-accelerated) + MTP head | |
| run: | | |
| mkdir -p models | |
| base="https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main" | |
| curl -fL -o models/gemma-4-12b-it-Q4_0.gguf "$base/gemma-4-12b-it-Q4_0.gguf" | |
| curl -fL -o models/mtp-gemma-4-12b-it.gguf "$base/mtp-gemma-4-12b-it.gguf" | |
| ls -lh models | |
| - name: Autotune draft length (Arm64 CPU) | |
| run: | | |
| python3 selfdraft/sd.py autotune models/gemma-4-12b-it-Q4_0.gguf \ | |
| --ngl 0 --threads "$(nproc)" --grid 1,2,3,4 --workload code --n-predict 160 \ | |
| --json bench_autotune.json | |
| - name: Bench self-draft - MTP only ($/1M tok) | |
| run: | | |
| python3 selfdraft/sd.py bench models/gemma-4-12b-it-Q4_0.gguf \ | |
| --ngl 0 --threads "$(nproc)" --n-max 3 --methods draft-mtp \ | |
| --workload code --n-predict 200 --price 0.0 --json bench_mtp.json | |
| - name: Bench self-draft - MTP + n-gram cascade | |
| run: | | |
| python3 selfdraft/sd.py bench models/gemma-4-12b-it-Q4_0.gguf \ | |
| --ngl 0 --threads "$(nproc)" --n-max 3 --methods draft-mtp,ngram-mod \ | |
| --workload code --n-predict 200 --price 0.0 --json bench_cascade.json | |
| - name: Upload results | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: arm64-bench-results | |
| path: | | |
| bench_autotune.json | |
| bench_mtp.json | |
| bench_cascade.json |