Popular repositories Loading

1. Annotated-LLM-Runtime Annotated-LLM-Runtime Public

   From-scratch, heavily-annotated CUDA inference runtime for Qwen2.5-Coder-7B on H100 (sm_90). Custom INT4 packer, fused GEMV, paged KV, split-KV attention, CUDA graph decode — every hot path comment…

   Cuda 21 4

2. WarpGroup-backend WarpGroup-backend Public

   A high-performance C++ backend for extreme-context LLM inference. It replaces item-count batching with dynamic, VRAM-aware First-Fit Decreasing (FFD) bin packing. By using PyBind11 for async queuei…

   Python 5 1

3. swarmkv swarmkv Public

   C++ inference runtime for llama.cpp that shares a single document KV-cache prefill across multiple analytical branches via snapshot fan-out. Eliminates redundant GPU compute and dramatically reduce…

   C++ 3 2

4. VRAM-Conductor VRAM-Conductor Public

   C++17 + CUDA orchestrator for running multiple LLM agents on one constrained GPU. `lmxd` daemon does NVML-seeded admission control to stop llama.cpp OOM crashes; `LayerStreamer` + `PinnedHostPool` …

   C++ 3

5. inter-llm-tokf inter-llm-tokf Public

   Inter-LLM knowledge handover framework leveraging Open Knowledge Format (OKF) for Qwen family. Bypasses text paring and tokenization by passing binary token arrays directly into model embedding lay…

   Python 3 2

6. sionna-munich-ai-ran sionna-munich-ai-ran Public

   GPU-accelerated Network Digital Twin for AI-RAN experiments in Munich. Powered by NVIDIA Sionna RT and PyTorch for ray-traced RF simulation. Includes real-world Munich simulation, Mitsuba 3 backend…

   Python 2 1