llm-cpu — a from-scratch CPU LLM inference engine

A transformer inference engine in C++17, built llama.cpp-style, CPU-first, for a Windows laptop with no GPU. It loads frozen weights and turns a prompt into text: tokenize → forward pass → sample → repeat.

This repo implements Phases 0–9 of The LLM Inference Engine Bible v2 (see docs/ARCHITECTURE.md for the code→ADR map), plus three extensions: min-p sampling, a 3-tier SIMD dispatch (scalar/AVX2/AVX-512), and SuperSpec multi-drafter speculative decoding.

Windows quick start (no build needed)

1. Download llm-cpu-windows-x64.zip from Releases and extract it. (Verify with the .sha256 file if you like.)
2. Download a supported Llama-family GGUF model separately — models are multi-gigabyte files with their own licenses and are never bundled. E.g. Llama-3.2-1B-Instruct-Q4_0.gguf from Hugging Face.
3. Run:

.\llm.exe --model .\Llama-3.2-1B-Instruct-Q4_0.gguf --chat `
  --prompt "Explain what a KV cache is." --n 128

Antivirus note. llm.exe is a new, unsigned open-source binary, so Windows Defender's cloud heuristic may occasionally flag a release download as a false positive (a ...!ml detection name means a machine-learning guess, not a signature match). The ZIP's .sha256 file lets you verify the download is exactly what CI built from the public source. If you prefer not to trust a downloaded binary at all — clone the repo and build it yourself in about a minute; the Build section below is the complete recipe, and the resulting exe is identical in function:

git clone https://github.com/lordisrael1/llm-cpu.git
cd llm-cpu
tools\build.cmd Release

Supported today	Not yet supported
Windows x64 (binary release)	macOS / Linux release binaries
Llama-family GGUF (Llama 3.x incl. tied embeddings)	Qwen, Mistral, Gemma, Phi architectures
F32, F16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, Q4_K, Q5_K, Q6_K tensors (incl. Q4_K_M files)	Q2_K / Q3_K, IQ-quants
Text generation + Llama-3 chat template	Vision, embeddings, batched serving

What's implemented

• Llama-family architecture: RMSNorm, RoPE (half-split), grouped-query attention (GQA), SwiGLU FFN, residual stream — validated against a numpy reference layer-by-layer (cosine > 0.999) with greedy output matching exactly.
• Hardened loaders (untrusted-input discipline, Section 21): a custom LEM1 format and a real GGUF v3 reader, both memory-mapped, with checked arithmetic, bounds-checked offsets, capped counts, and fail-closed validation.
• Quantization: Q8_0 and Q4_0 with dequantize-on-the-fly (GGUF-compatible block layouts). Q8 cosine > 0.999, Q4 > 0.95 vs the F32 reference. K-quants (Q4_K / Q5_K / Q6_K) for real GGUFs — Q4_K_M, the most common quant on Hugging Face, loads and runs.
• Perplexity harness (--perplexity FILE): chunked next-token perplexity over a text file, for objective quality comparison across quant levels.
• Performance: persistent thread pool (parallel over output rows), AVX2 FMA dot product with runtime CPUID dispatch, scalar fallback. Bit-identical output across thread counts (determinism = continuous race detector).
• Generation: preallocated KV cache, prefill/decode loop, an explicit runtime state machine, memory preflight, and a pluggable sampler (greedy / temperature / top-k / top-p / min-p, seeded RNG).
• SuperSpec speculative decoding (multi-drafter, adaptive-K); see below.
• Safe CLI: streaming output with terminal sanitization, stdout=data / stderr=commentary, UTF-8 console, NO_COLOR, and an exit-code taxonomy.

100 unit tests, all passing and clean under AddressSanitizer.

Build (Windows, MSVC)

Requires Visual Studio Build Tools 2022 (MSVC + the Windows 11 SDK component) and Python 3 with numpy (build-time only, for the reference model).

:: generate the tiny reference model, tokenizer, and quantized/GGUF variants
python tools\export_weights.py
python tools\dump_reference.py
python tools\export_tokenizer.py
python tools\export_quantized.py
python tools\export_gguf.py
python tools\export_q6k_test.py

:: build (Release) and run tests
tools\build.cmd Release
build\Release\unit_tests.exe

:: build + run the full suite under AddressSanitizer
tools\build.cmd Debug ASAN
build\Debug-asan\unit_tests.exe

tools\build.cmd invokes vcvars64.bat and the CMake/Ninja bundled with Build Tools, so no separate CMake install is needed.

Run

build\Release\llm.exe --model models\tiny.lem --prompt "Hello" --n 32
build\Release\llm.exe --model models\tiny.f32.gguf --prompt "Hello" --n 32

:: SuperSpec: F32 target verifying Q8 + Q4 drafters (output == plain greedy)
build\Release\llm.exe --model models\tiny.lem ^
  --spec models\tiny.q8.lem --spec models\tiny.q4.lem --prompt "Hello" --n 32

Generated text goes to stdout; all stats/warnings go to stderr, so llm ... > out.txt yields clean text. Run llm --help for all flags.

For an instruct model, use --chat so it answers and stops instead of continuing the document ("continuation bias"):

llm --model Llama-3.2-1B-Instruct-Q4_0.gguf --chat --prompt "What is the capital of France?" --n 64
llm --model model.gguf --chat --system "You are terse." --prompt "..." --n 64

--chat wraps the prompt in the Llama-3 template (<|begin_of_text|>, <|start_header_id|>, …), injecting those special tokens by id (not BPE-encoded), and stops at <|eot_id|>.

Note: the bundled models/tiny.lem has random, untrained weights (it exists to validate the engine, not to produce meaningful language). Output is therefore gibberish — expected. Point the loader at a trained GGUF with a matching tokenizer to get real text.

Running a real model from HuggingFace

The engine reads only local files — no network, no API keys. To try a real model you download a GGUF in your browser and point --model at it. The GGUF reader now also parses the model's embedded byte-level BPE tokenizer (tokenizer.ggml.*), so no separate --tokenizer is needed for gpt2/llama-bpe models.

Which model: pick one whose architecture matches this engine — Llama-family (RMSNorm, RoPE, GQA, SwiGLU, no attention biases) with a byte-level BPE tokenizer. The best fit:

Llama-3.2-1B-Instruct (GGUF, Q4_K_M, Q8_0, or Q4_0) — e.g. the bartowski/Llama-3.2-1B-Instruct-GGUF repo on HuggingFace. Small enough for a laptop CPU; llama architecture; byte-level BPE tokenizer.

Avoid for now: Qwen2 (has attention biases this engine doesn't implement), TinyLlama / Llama-2 (SentencePiece tokenizer, not byte-level BPE), and anything non-llama.

Status — the main architectural blockers are handled:

1. ✅ Tokenizer — byte-level BPE is extracted from the GGUF and used, with the llama-bpe pre-tokenizer regex applied (validated against a Python regex reference with 0 mismatches on a 1000-case differential fuzz).
2. ✅ RoPE weight permutation — the loader un-permutes Q/K rows back to the HF layout on load (llama.cpp permutes them for its interleaved RoPE; this engine uses half-split RoPE). Verified byte-exact against the un-permuted reference, dtype-agnostic (works on quantized rows).
3. ✅ Tied embeddings — models without a separate output.weight (e.g. Llama-3.2-1B) fall back to the token embedding for the final projection.

Remaining caveat for exact parity (won't stop it running, may affect quality): Llama-3's RoPE scaling (rope_scaling: llama3) isn't applied, so very long contexts drift. End-to-end verified on real GGUFs: Q4_0, Q4_K_M, and Q8_0 builds of Llama-3.2-1B-Instruct all load, chat correctly, and stop at <|eot_id|> (see the perplexity table below for measured quality).

Benchmarks (this machine)

Quantization quality (perplexity, lower is better)

Llama-3.2-1B-Instruct, first 8 KB of the wikitext-2-raw test set (1,855 scored tokens, 512-token windows), 12 threads, AVX2:

Quant	File size	Perplexity	nats/token
Q4_0	773 MB	19.90	2.991
Q4_K_M	808 MB	18.62	2.924
Q8_0	1.32 GB	17.86	2.883

Q4_K_M closes ~63% of the quality gap between Q4_0 and the near-lossless Q8_0 baseline for only ~4.5% more bytes — the measured payoff of the K-quant per-sub-block scale+min encoding. Reproduce with:

llm --model <model.gguf> --perplexity wiki.test.raw

(The number is a subset measurement for laptop-CPU runtimes, not comparable to full-corpus figures from GPU rigs; the relative ordering is the point.)

Throughput

Measured on the tiny model (dim=64, 2 layers, vocab=288), 12 threads, AVX2:

Path	tok/s (approx)
Plain greedy	~4000
GGUF F32	~4000
SuperSpec (2 drafters)	~700

These numbers are dominated by per-token overhead on a deliberately tiny model; they demonstrate the engine runs, not datacenter throughput. SuperSpec is slower here on purpose — see its note below. On a real quantized 1B model expect single- to low-double-digit tok/s on a laptop CPU, which is normal and usable.

SuperSpec (multi-drafter speculative decoding)

Adapted from C. Shen, R. Guo, Y. Cheng, Y. Lin, Z. Zhao, Y. Hu, S. Chen, X. Liu, K. Li, "SuperSpec: Enhanced Verification and Sampling for End-to-End LLM Speculative Decoding," IEEE HPCC 2025 (DOI 10.1109/HPCC67675.2025.00064, Tianjin University).

This engine implements SuperSpec's architecture and algorithms: multiple drafters each with their own KV cache, an array-based batch verifier (not a token tree), a global-optimal sampler (commit the longest accepted prefix), Algorithm 1 adaptive-K / bubble controller, KV rollback via cache rewind, and per-round metrics.

Correctness invariant (tested): with greedy verification, committed tokens are always the target's greedy continuation, so SuperSpec output is bit-identical to plain greedy target output for any drafter set or K. Multi-drafter and adaptive-K change speed, never the result.

Honest limitation: the paper's wall-clock speedup comes from a GPU batched verifier forward (K positions in one pass) and custom CUDA/Triton kernels — which the paper does not publish. This engine decodes one token per call on CPU, so what carries over is the architecture, algorithms, and correctness, not the throughput (hence SuperSpec is slower here). The paper flags this gap itself.

Initial-Key Cache (selective KV, bounded long-context memory)

Adapted from Z. Tang, Z. He, J. Shen, Y. Hu, L. Zhou, Y. Nie, Y. Wang, "Initial-Key Cache," IEEE IJCNN 2025 (DOI 10.1109/IJCNN64981.2025.11228214, National University of Defense Technology).

Instead of caching every token's KV, keep three groups — initial (attention sinks), recent (local context), and the top-k important middle tokens by "Score Sum" (a rolling sum of the last few attention rows, reusing softmax weights already computed, so zero extra transformer FLOPs) — and evict + compact the rest when the cache exceeds a budget. This gives long context with bounded KV memory.

:: cap the live KV cache at 256 slots regardless of sequence length
llm --model model.gguf --prompt "..." --n 2000 --kv-budget 256 --kv-recent 128

Correctness: with --kv-budget >= ctx (or a budget larger than the run) the policy falls back to / matches full attention exactly — verified against the reference greedy output. Note: this engine uses global eviction across layers; the paper specifies per-layer scores (a documented simplification).

Layout

See docs/ARCHITECTURE.md. Core library in src/ + include/engine/, thin CLI in app/, build-time Python tooling in tools/, tests in tests/, models (gitignored) in models/.