[NeurIPS 2025] NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
Production-oriented LLM serving examples covering KV-cache decoding, batching, quantization, LoRA, multi-LoRA, testing, benchmarking, and reproducible MLOps workflows.
Transparent LLM inference experiments on Apple Metal
Optimizing LLM throughput via binned padding, sequence packing, and Flash Attention.
This project replaces broken tokenization heuristics for Roman Urdu-English text with a single-feature Ridge model, achieving 2.5× higher accuracy at an ultra-low latency of 0.003 ms. By eliminating subword underestimation, our ML-driven scheduler drops misrouted requests to 3.4%, cutting queue wait times by 28% over standard word heuristics.
Layer-wise weight offloading with profiling-driven optimizations.
QTIP and QUIP Quantization from First Principles. Most people understand basic INT8 quantization, but don't know exactly why lower bit-widths like INT4 completely destroy model quality, and how we fix it. To understand how we can run massive models on consumer hardware, we need to understand the evolution of quantization up to QTIP.
Add a description, image, and links to the llm-inference-optimization topic page so that developers can more easily learn about it.
To associate your repository with the llm-inference-optimization topic, visit your repo's landing page and select "manage topics."