Add llama.cpp backend (GGUF) for raw / L0 / L1

[Tusm11]

Closes Issue #1

Adds LlamaCppBackend (anyjev/backends/llamacpp.py), the llama.cpp slot from the ROADMAP's help-wanted list. It runs GGUF models through llama-cpp-python at raw / L0 / L1.

What it does

• Last-row logits only. Each prompt is decoded from a cleared KV cache (llama_memory_clear). Only the final token is flagged for output in the llama_batch, and its row comes from llama_get_logits_ith(-1). There is no logits_all. The backend returns the full-vocabulary log-softmax of that row at the requested ids. Nothing is sampled.
• Tokenizer.
  • By default, the GGUF's own vocabulary and chat template are used, so transformers is not needed.
  • With tokenizer= (a Hugging Face name or object), prompts are rendered exactly as HFBackend renders them.
  • In that mode, every prompt is tokenized by both sides and must match token for token, and every label id must name the same text in both vocabularies. Otherwise the backend raises TokenMismatchError.
• Install. pip install "anyjev[llamacpp]" installs llama-cpp-python>=0.3.16. Older versions are refused with a clear message.
• Scope. There is no hidden_states_to, so no L2, and no score_shared. Both could be follow-ups.

Files

• anyjev/backends/llamacpp.py (new)
• scripts/llamacpp_parity.py (new)
• tests/test_llamacpp.py (new): 12 tests on a stub llama_cpp module, which run in CI, plus one @pytest.mark.engine smoke test that runs when ANYJEV_GGUF points to a GGUF file
• pyproject.toml: adds the llamacpp extra and registers the engine marker
• CHANGELOG.md: one line

Results (all runs on my machine; the commands are below)

Parity against HFBackend. Qwen2.5-0.5B-Instruct, transformers in float32 on CPU, 15 prompts (choice / noul / score), same rendered prompts and label ids on both sides:

GGUF	argmax agreement	max |Δ logprob|	mean |Δ logprob|
F32, converted from the same HF snapshot (convert_hf_to_gguf.py --outtype f32)	15/15	0.0159	0.0089
Official Qwen/Qwen2.5-0.5B-Instruct-GGUF fp16	14/15	1.1692	0.7053

The F16 gap comes from llama.cpp computing with the F16 weights, not from the backend. On the same F16 file:

• the backend matches llama-cpp-python's own Llama.eval(logits_all=True) last row to max |Δ| 0.0002 (15/15)
• an F32 KV cache changes the result by only 0.04

The parity script's docstring therefore recommends an F32 conversion.

Engine smoke test. pytest -m engine passes on Qwen2.5-7B-Instruct Q8_0.

CI. ruff check anyjev bench demo handoff scripts space tests && pytest -q passes.

Environment: Windows, CPU, Python 3.13, llama-cpp-python , torch , transformers . The versions are in env in the parity JSON.

python scripts/llamacpp_parity.py --gguf qwen2.5-0.5b-instruct-f32.gguf --model Qwen/Qwen2.5-0.5B-Instruct --device cpu --dtype float32 --json parity_f32.json
python scripts/llamacpp_parity.py --gguf qwen2.5-0.5b-instruct-fp16.gguf --model Qwen/Qwen2.5-0.5B-Instruct --device cpu --dtype float32 --json parity.json

I haven't committed the parity JSONs, since vLLM's parity script doesn't commit its output either. I can add them under bench/ if you'd like.

Side note (not changed here)

HFBackend passes device_map=, which needs accelerate, but the hf extra doesn't list it. I had to pip install accelerate to run the parity script. I can open a separate one-line PR if that's useful.

[Tusm11]

Saw @MorrisZJ comment on the issue about the serving-path rework after I'd already opened this, sorry for the timing. I'll rebase once you confirm main has settled.

[MorrisZJ]

Saw @MorrisZJ comment on the issue about the serving-path rework after I'd already opened this, sorry for the timing. I'll rebase once you confirm main has settled.

@Tusm11 Thanks for the contribution, and no worries about the timing! We just landed the serving-path rework, so main should be stable now. Could you rebase onto the latest main and refactor this PR to follow the new method? Feel free to ping me if anything in the new interface is unclear.

[Tusm11]

Saw @MorrisZJ comment on the issue about the serving-path rework after I'd already opened this, sorry for the timing. I'll rebase once you confirm main has settled.

@Tusm11 Thanks for the contribution, and no worries about the timing! We just landed the serving-path rework, so main should be stable now. Could you rebase onto the latest main and refactor this PR to follow the new method? Feel free to ping me if anything in the new interface is unclear.

Rebased onto the current main. The rework (transformers-5 compat fix in HFBackend, L2 support in VLLMBackend) doesn't touch the Backend protocol this PR implements, so no functional changes were needed. I did update the docstring to state the raw/L0/L1 vs. no-L2 boundary explicitly, matching the style in the updated vllm.py. Re-ran the full suite (103 passed, 1 skipped), the engine test on a real 7B model, and the parity script against transformers. same result as before, 15/15 argmax agreement with the F32 GGUF. Ready for review.

[monke-sniper]

Independent check from a Windows box — i7-14700F, RTX 4060 8 GB, Python 3.13, llama-cpp-python 0.3.35 (default PyPI wheel), transformers 4.57.6:

• pytest -q on this branch: 103 passed, 1 skipped.
• scripts/llamacpp_parity.py on an f32 GGUF converted from Qwen/Qwen3-0.6B (--device cpu --dtype float32): 15/15 argmax agreement, max |Δ| full-vocab log-prob 0.0352 / restricted log-softmax 0.0326 (mean ~0.016). Reproduces the number in the description.
• Engine smoke test (ANYJEV_GGUF=<gguf>) passes, and a real Ollama blob (qwen3:4b, Q4_K_M, 151k vocab, chat template read from the GGUF) loads in-process and answers a Decider call — first decision (raw, one prefill, CPU) in 0.77 s.

One environment note that may be worth a line for Windows users: the cu124 wheel from the usual extra-index crashes here at context creation — OSError: [WinError -1073741795] Windows Error 0xc000001d (illegal instruction) at llama_cpp.llama_init_from_model. That wheel's llama_print_system_info() reports AVX512 = 1 on a CPU that has only avx/avx2 (Raptor Lake ships no AVX-512), while the default PyPI wheel prints no AVX-512 line and runs fine. It looks like the prebuilt CUDA wheel's codegen rather than this backend; "use the default wheel, or build with -DGGML_NATIVE=OFF, if the CUDA wheel traps" would be a useful pointer in the docstring.

FWIW the same GGUF reads correctly through a llama-server build (PrismML b10660) on this machine too, so it is not the model file.

[craftingmod]

I made a small follow-up PR against your branch to support reusing an existing Llama/Llm instance and compatible high-level logits APIs. Feel free to view if it fits the scope of this. 👀

[MorrisZJ]

Thanks @Tusm11, and sorry for the wait — I wanted to run this rather than just read it. Thanks also to @monke-sniper for the independent reproduction and to @craftingmod for the follow-up.

Everything below was run against this branch on Linux with llama-cpp-python 0.3.35 and Phi-3-mini-4k-instruct Q4 (a SentencePiece vocabulary, chat template read from the GGUF).

What holds up

The hard part is right. I checked the chunked decode, the absolute positions across chunks, the single output flag on the last token of the last chunk, and llama_get_logits_ith(ctx, -1); every llama.cpp symbol you call exists at your 0.3.16 floor with a matching signature. Tests, a parity script and an engine smoke test is exactly the shape the help-wanted slot asked for.

And one concern of mine turned out to be wrong, which is worth reporting explicitly. I expected that on a SentencePiece vocabulary GGUFTokenizer.encode would resolve a bare label to its leading-space twin, so the backend would read a logit the model never emits — with _check_label_ids unable to catch it, since it is a no-op when no HF tokenizer is given. On Phi-3-mini, encode("A") does return the space-prefixed id 319 — but that is exactly the token the model emits after <|assistant|>\n: rank 1 at p = 1.0000, with the bare A (29909) at rank 6 and no mass. Across choice, noul and score the probability mass landing on the resolved label ids was 1.0000 / 0.9998 / 1.0000, and every answer was sensible. So the label path is correct on the family I was most worried about. Worth recording because both reproductions so far used byte-level BPE models, which would not have exercised it either way.

What I would like changed before merge

name is a filename, and it is the artifact identity key. backend.name is what Decider.load_artifact / load_artifacts check. llama-quantize writes ggml-model-Q4_K_M.gguf by default, so two different models in different directories collide. Measured:

backend A name: 'ggml-model-Q4_K_M'   (.../modelA/ggml-model-Q4_K_M.gguf)
backend B name: 'ggml-model-Q4_K_M'   (.../modelB/ggml-model-Q4_K_M.gguf)
L1 artifact fit on A -> load into B: ACCEPTED with no error

That artifact carries a frozen temperature (5.214 here) and a frozen prior, so it silently miscalibrates the other model. The inverse hurts the workflow this backend most plausibly exists for — calibrate on a GPU, ship the GGUF:

artifact fit through HFBackend("microsoft/Phi-3-mini-4k-instruct")
  -> refused: artifact was fit on microsoft/Phi-3-mini-4k-instruct, backend is ggml-model-Q4_K_M

llm.metadata already gives you general.basename / general.name; defaulting name from those fixes both directions. Worth a line in the docstring saying name= is the artifact key rather than a label.

Use after close() is a segfault, not an exception. close() frees the batch and the Llama but leaves self.llm bound and adds no closed-state guard, so the next call reaches _clear_memory() and hands a NULL context to llama_get_memory. In a subprocess:

closed; calling again...
exit code -11        <-- SIGSEGV, no traceback

Setting self.llm = None in close() and raising a RuntimeError at the top of next_token_logprobs turns that into something diagnosable.

next_token_logprobs truncates silently. zip(prompts, token_ids) stops at the shorter one: next_token_logprobs(['a','b','c'], [[319],[350]]) returns 2 rows with no error, where HFBackend indexes token_ids[i] and raises IndexError at the call site. Decider._score then leaves out[i] = None and the failure surfaces far from the cause. ids is also iterated twice — once by _check_label_ids, once by list(ids) — so a generator is drained by the check and every prompt comes back empty.

The version gate is wrong in both directions. Running it as written:

0.3.16rc1    -> (0, 3)  -> REFUSED: "needs llama-cpp-python>=0.3.16, found 0.3.16rc1"
0.3.16+cu124 -> (0, 3)  -> REFUSED
0.a.20       -> (0, 20) -> accepted
v0.3.9       -> (3, 9)  -> accepted

isdigit() drops a non-numeric component instead of truncating at it, and the two it refuses are exactly the prebuilt CUDA/Metal wheels most GGUF users install — quite possibly related to what @monke-sniper ran into. Since llama_get_memory / llama_memory_clear / llama_batch_init are plain module attributes, hasattr tests the real precondition and can name the missing symbol; that is also the rule hf.py:28-33 writes down for itself.

Decider.observe() fails late. It accepts 29 labelled states and raises on the 30th, when the first fit_head is attempted: ValueError: backend phi3-mini-q4 exposes no hidden states; L2 needs a local transformers backend. level="auto" and level="L2" both degrade cleanly, so this is the one rough edge — worth a sentence in the class docstring that the label-collection workflow cannot complete on this backend. (Small correction to the CHANGELOG line while you are there: the disqualifier is the missing hidden_states, not hidden_states_to — VLLMBackend has no hidden_states_to either and does serve L2.)

The chat template cannot compile {% generation %}. The Jinja environment registers loopcontrols but not the assistant-tracking extension, so a template carrying that tag dies with TemplateSyntaxError: Encountered unknown tag 'generation'. To be fair about the blast radius: of the templates I checked only SmolLM3-3B carries the real tag — Qwen3-0.6B/4B, Qwen2.5-7B, Phi-3-mini, Mistral-v0.3 and SmolLM2 do not, which is why nobody has hit it. The reason to take it anyway is that the fix is less code than what is there: llama_cpp.llama_chat_format.Jinja2ChatFormatter is the same sandbox with the same filters, Llama.__init__ has already built one from the same metadata key, and it registers IgnoreGenerationTags. That also gets you tools / documents binding (a template guarded on {% if tools is not none %} renders differently under your environment than under transformers, because Undefined is none is False), removes a per-prompt template recompile, and drops the jinja2 import that tests/test_llamacpp.py:223 concedes is llama-cpp-python's dependency rather than ours.

Performance — follow-ups, except one line

• llama_memory_clear(mem, True) memsets the whole KV data buffer on top of resetting the cells, once per prompt. Every prompt decodes from position 0 into fresh cells, so the data is never read; data=False is sufficient. At the default n_ctx=4096 on a 7-8B GGUF that is roughly half a gigabyte zeroed per prompt. One word.
• There is no prefix reuse: the K cyclic shifts of one state each re-decode the identical state prefix, on the engine where reuse is cheapest and the CPU slowest, and shared_prefix is silently a no-op with stats['shared_groups'] stuck at 0. Happy to take that as a second PR — but please drop the clears == 3 assertion (tests/test_llamacpp.py:244) in this one, so the follow-up does not have to fight the suite.

Housekeeping

One structural note: the default path — no tokenizer=, the one the module docstring recommends — is also the only one CI never exercises, because its single test starts with pytest.importorskip("jinja2") and jinja2 is in neither the dev nor the llamacpp extra. Reusing the library's formatter removes the import and the gap together.

Yes please to committing the parity JSONs — AGENTS.md rule 1 is that every number in prose names its JSON, and the docstring's "~1.2 nats" / "~0.02" name none (that is also why vllm_parity.py quotes no numbers). @monke-sniper's Windows note about the cu124 wheel trapping in llama_init_from_model is worth a line there too.

I will add you to CREDITS.md with the issue number on the CHANGELOG line when this lands — our side, not yours.

Separately: yes, please open that one-line PR adding accelerate to the hf extra. Someone reported the same hole from the other direction in #5; we have just closed it by dropping device_map instead, but the catch was good.

@craftingmod — reusing an externally created Llama is a sensible addition, and the high-level-API path is close to what I would have suggested anyway. It does interact with the close() question above: if the instance comes from outside, this backend should not be the one freeing it. Worth settling that ownership rule as part of it.

[Tusm11]

Thanks @monke-sniper for the independent reproduction and the Windows/cu124 report, and @MorrisZJ for actually running this.

Pushed fixes for everything flagged:

• name now defaults from the GGUF's own general.basename/general.name metadata instead of the filename. llama-quantize writes ggml-model-Q4_K_M.gguf for every model alike, so different quantized models could previously collide on the same default artifact-identity key. Explicit name= still overrides this.
• Using the object after close() now raises RuntimeError instead of potentially segfaulting. close() sets self.llm = None, and every method checks this first.
• next_token_logprobs no longer silently truncates on length mismatch, and label IDs are only iterated once. A generator was previously consumed by the validation check and then came back empty.
• The llama-cpp-python version gate now checks required symbols with hasattr instead of parsing __version__, fixing the 0.3.16+cu124-style rejection reported by @monke-sniper.
• Chat templates now render through llama_cpp.llama_chat_format.Jinja2ChatFormatter, the same formatter that Llama() itself builds. This fixes {% generation %}-tagged templates such as SmolLM3's that previously raised TemplateSyntaxError. The formatter is built once per add_generation_prompt value rather than recompiled on every call.
• _clear_memory now passes data=False to llama_memory_clear instead of True.
• Dropped the clears == 3 assertion so a future prefix-reuse PR does not have to fight this test suite.
• Corrected the CHANGELOG: the L2 disqualifier is missing hidden_states, not hidden_states_to.
• Updated the class docstring to document the Decider.observe() / L2 rough edge.
• Added parity JSONs at bench/llamacpp_parity_f32.json and bench/llamacpp_parity_f16.json, cited by name in the docstring instead of the old unsourced numbers, along with the Windows/cu124 note.
• Added regression tests covering all of the above.

Validation:

• pytest -q: 108 passed, 1 skipped
• pytest -m engine: passes against a real 7B GGUF

Intentionally not touched in this PR:

• @craftingmod's external-Llama-reuse PR, since it interacts with the close() ownership question and is better handled separately.
• Prefix reuse/batching.

Thanks again for the reports and testing.

[MorrisZJ]

Thanks @Tusm11, and sorry this sat for a while. 0.3.0 landed in between. I re-checked your branch on top of the current main (0.3.0 plus the evaluation commit): with your files applied, ruff is clean and the whole suite passes (100 passed, 9 skipped), including all 16 of your stub tests. Every point from the last review is addressed. Thank you, this is ready apart from the 0.3.0 changes.

0.3.0 removed the label-trained levels (L1 temperature artifacts, L2 heads, observe, level="auto"), so this backend is now simply raw / L0. Could you rebase and adjust a few things?

Rebase onto main. pyproject.toml conflicts because the bench extra is gone; put llamacpp = ["llama-cpp-python>=0.3.16"] next to vllm / client / eval and keep your markers line. In CHANGELOG.md, move your line under ## Unreleased and say raw / L0.
Docstrings. In the module docstring, change "(L0 / L1)" to "(raw / L0)". In the class docstring, drop the paragraphs about load_artifact / L1 calibration and observe() / L2. Decider no longer reads backend.name, so name is only a display name now (keeping the metadata default is still the right call).
tests/test_llamacpp.py: the comment in test_name_defaults_from_gguf_metadata_over_the_filename about L1 artifacts can go as well.
Parity JSONs: please record the GGUF's file name rather than the local path (D:\models1...).
Optional: add jinja2 to the dev extra, so CI exercises the default GGUF-template path; that test is currently skipped there.

Once that's in, we'll merge. On our side we'll add you to CREDITS.md with #1 on the CHANGELOG line, and take llama.cpp off the help-wanted lists.

[Tusm11]

Thanks @MorrisZJ , rebased onto the current main (0.3.0 plus the evaluation commit) as a single commit. One note: the PR was briefly closed by my own force-push while I was rebasing, and I've reopened it.

Done as requested:

-> pyproject.toml: llamacpp = ["llama-cpp-python>=0.3.16"] is next to vllm / client / eval, and the engine markers line is kept. I added jinja2 to the dev extra so CI exercises the default GGUF-template path.
-> CHANGELOG.md: the entry is under ## Unreleased and says raw / L0.
-> Docstrings: the module docstring now says "(raw / L0)". The class docstring no longer has the load_artifact / L1 and observe() / L2 paragraphs. name is described as a display name, keeping the metadata default.
-> tests/test_llamacpp.py: the L1-artifact comment is removed.
-> Parity JSONs now record the GGUF's file name instead of the local path, and scripts/llamacpp_parity.py writes the file name from now on.

ruff check anyjev scripts space tests is clean, and pytest -q gives 109 passed, 1 skipped on Windows (the skip is the engine test). Separately, tests/test_readme_numbers.py calls read_text() without an encoding, so on Windows it fails with a UnicodeDecodeError unless PYTHONUTF8=1 is set. Adding encoding="utf-8" there would fix it. I left it out of this PR to keep it to one change.

[Tusm11]

Thanks @MorrisZJ for the thorough reviews and for merging, and @monke-sniper and @craftingmod for testing it on your machines. Happy to help if anything turns up on other platforms.