🛠️[v0.3.45] Release Note: Reactivated Built-in Embeddings, Modern Model Loading, and Stronger Cross-Platform Reliability

[JamePeng]

v0.3.45

Reactivated Built-in Embeddings, Modern Model Loading, and Stronger Cross-Platform Reliability

I’m pleased to release llama-cpp-python v0.3.45, focused on reactivating and modernizing Llama’s built-in embedding capabilities, aligning the Python bindings with the latest llama.cpp APIs, and improving reliability across platforms.

Highlights

Refactored and Reactivated Built-in Embeddings

I have refactored and reactivated the embedding functionality built directly into the standard Llama class. The maintained embed() and create_embedding() APIs now use sequence-aware streaming batches based on the current LlamaBatch interface.

The updated implementation supports:

• Strings, batches of strings, and pre-tokenized inputs
• Sentence-level and token-level embeddings
• Rank pooling for reranking models
• Separator-based input splitting
• Token accounting and OpenAI-compatible responses
• Boolean and llama.cpp-compatible normalization modes
• Streaming governed correctly by n_seq_max

This release also fixes invalid sequence ID errors in LlamaEmbedding when processing multiple documents with the default n_seq_max=1 configuration. Batch validation errors now provide clearer, actionable guidance for configuring n_seq_max, n_batch, and n_ubatch.

I have substantially expanded the embedding and reranking documentation, including end-to-end examples, pooling and normalization guidance, output shapes, batching strategies, supported model families, common configuration issues, and resource-safe usage patterns.

Updated Model Loading Interface

I have aligned model loading with the latest llama_model_params API by introducing load_mode.

It supports:

• LLAMA_LOAD_MODE_NONE
• LLAMA_LOAD_MODE_MMAP
• LLAMA_LOAD_MODE_MLOCK
• LLAMA_LOAD_MODE_MMAP_MLOCK
• LLAMA_LOAD_MODE_DIRECT_IO

The legacy use_mmap, use_direct_io, and use_mlock options are now deprecated. They remain accepted for migration compatibility and produce a warning when used.

I have also exposed the new no_alloc and load_mtp model-loading options and enabled extra buffer types by default to support weight repacking and the latest llama.cpp loading behavior.

Latest llama.cpp and Binding Synchronization

I updated the bundled llama.cpp revision to [876a432](https://github.com/ggml-org/llama.cpp/commit/876a4321163249c43ca4e986818fab5ab081f282) and synchronized the llama, mtmd, and ggml bindings with the upstream APIs as of August 1, 2026.

The optional llama_ext ctypes bindings now use the correct signatures, including unsigned 32-bit layer IDs, the correct void return type for embedding extraction controls, and the correct target-layer count return type.

Cross-Platform ABI Compatibility

I added a cross-platform ABI inspection utility for PE, ELF, and Mach-O binaries. It can normalize platform-specific exported symbol names and validate optional llama_ext ctypes aliases across Windows, Linux, and macOS builds.

Optional extension lookup now also recognizes GCC and Clang Itanium C++ ABI symbol variants, improving compatibility on Linux and macOS. Thanks to @ckcfcc for reporting the related issue.

Reliability and Resource Management

I strengthened cleanup behavior when model, batch, context, or sampler initialization fails. Native resources are now registered for ownership immediately after allocation and released exactly once, including during partially completed initialization.

Closed wrappers also clear their parent model, vocabulary, and sampling references, preventing unnecessary object retention. New failure-injection and lifecycle tests cover these paths and verify that repeated cleanup remains safe.

On Windows, stale HIP_PATH or VULKAN_SDK environment variables no longer cause import llama_cpp to fail when their directories have been removed. Thanks to @emptyngton for contributing this fix.

Tests, Documentation, and Branding

I refreshed the chat-format test suite with self-contained Qwen3.5-style templates covering thinking modes, multimodal placeholders, tools, tool-call history, invalid inputs, and model-specific stop tokens.

This release also includes:

• Expanded Llama API and server documentation
• Updated embedding examples and performance-tuning guidance
• A comprehensive embeddings and reranking guide
• New ABI inspection documentation and tests
• Updated project branding and a repository-local logo

For the complete diff, see [v0.3.44…v0.3.45](ebf6099...aafc6fb).

— JamePeng