Fix broken bindings and add missing llama.cpp API coverage with examples - #4
Merged
Merged
Conversation
- Fix batch struct size bug: allocate 56 bytes (not 48) for llama_batch struct to correctly account for all 6 pointer fields after n_tokens - Add missing FFI declarations: grammar sampling, logit bias, infill/FIM, LoRA adapters, state save/load, pooling type, vocab score/attr, model quantization - Add high-level wrappers for all new bindings with ergonomic APIs - Add constants for token attributes and quantization types - Add 7 new examples demonstrating different capabilities: - embeddings.hml: text embeddings with cosine similarity - chat_template.hml: proper chat template formatting - grammar.hml: GBNF grammar-constrained JSON generation - tokenizer.hml: vocabulary and tokenization exploration - streaming.hml: streaming generation with performance metrics - infill.hml: code fill-in-the-middle (FIM) completion - model_info.hml: model metadata and system inspection - Update README with new API reference and example documentation https://claude.ai/code/session_01PQRPJWjp2HfT41sKmVaSsy
- Expand test_basic.hml with 40+ assertions covering constants, sampler creation/chain/introspection, and backend functions (all without requiring a model file) - Add test_model.hml with comprehensive model-dependent tests: model info, vocab, tokenization round-trips, decode, sampling, KV cache, chat templates, grammar, logit bias, generation - Add scripts/download_test_model.sh to fetch a ~15MB TinyStories model from HuggingFace for quick testing - Add models/ to .gitignore - Update CI to validate all new example files https://claude.ai/code/session_01PQRPJWjp2HfT41sKmVaSsy
- Split test step into basic tests (no model) and model tests - Download ~15MB TinyStories Q4_0 model in CI for model-dependent tests - Model download and tests only run when Hemlock compiler is available - Lint job also checks download_test_model.sh syntax https://claude.ai/code/session_01PQRPJWjp2HfT41sKmVaSsy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
struct to correctly account for all 6 pointer fields after n_tokens
LoRA adapters, state save/load, pooling type, vocab score/attr,
model quantization
https://claude.ai/code/session_01PQRPJWjp2HfT41sKmVaSsy