Local Inference Lab

Popular repositories Loading

1. rtx6kpro rtx6kpro Public

   RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink

   Python 1.2k 92

2. b12x b12x Public archive

   Archived: b12x development has moved to FlashInfer. See README for migration and PR-porting guidance.

   Python 269 70

3. llm-inference-bench llm-inference-bench Public

   LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.

   Python 122 23

4. blackwell-llm-docker blackwell-llm-docker Public

   Docker images for LLM inference (SGLang + vLLM) on NVIDIA Blackwell GPUs (SM120, CUDA 13.2)

   Python 97 27

5. vllm vllm Public

   Forked from vllm-project/vllm

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 44 43

6. quant-toolkit quant-toolkit Public

   Python 17 9