Skip to content

fine_tuning_guide.md

Latest commit

 

History

History
49 lines (36 loc) · 2.56 KB

fine_tuning_guide.md

File metadata and controls

49 lines (36 loc) · 2.56 KB
Raw

BitNet Optimization & Fine-Tuning Guide

1. Performance Optimization (Latency)

Your current ~30s latency is due to the Context Window Size. By default, RAG pipelines retrieve large chunks of text. Local CPUs struggle to process thousands of tokens quickly.

What We Did (Immediate Fix)

We updated the backend to reduce the context size for local models:

  • Cloud (Gemini): Retrieves 5 chunks (high context).
  • Local (BitNet): Retrieves 2 chunks (focused context).

Further Optimizations

  1. Reduce Chunk Size: currently 1000 chars. In services/document_processor.py, try reducing this to 500.
  2. Use llama-server Caching: Ensure your server is started with -c 2048 (context size) and prompt caching enabled (default in recent versions).
  3. Hardware Acceleration: If you have an NVIDIA GPU, ensure llama-server is compiled with CUDA support (make LLAMA_CUDA=1).

2. Fine-Tuning BitNet

Warning: Fine-tuning actual 1.58-bit weights is extremely cutting-edge and difficult. Standard tools (LoRA/QLoRA) often don't work well directly on 1-bit weights because gradients vanish.

Recommended Approach: "Distillation via Llama 3"

Instead of training BitNet directly, you assume Llama 3 8B is your teacher.

  1. Prepare Dataset:

    • Create a JSONL file with Question-Answer pairs relevant to your college/docs.
    • Format: {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

  2. Fine-Tune a Small Llama Model:

    • Use Unsloth (highly recommended for speed).
    • Fine-tune Llama-3-8B or Llama-3.2-3B (smaller/faster).
    • Export to GGUF using llama.cpp.

  3. Quantize to BitNet-like Precision:

    • You cannot easily convert a standard model into a true BitNet b1.58 architecture without retraining from scratch.
    • Best Alternative: Quantize your fine-tuned Llama 3 model to 2-bit (Q2_K) using llama.cpp.
    ./llama-quantize my-finetuned-model.gguf my-model-q2_k.gguf Q2_K
    • This gives you speed very close to BitNet while easier to train.

True BitNet Training (Advanced)

If you must train a BitNet b1.58 from scratch:

  • You need the BitNet b1.58 training code.
  • This requires massive compute (H100/A100 GPUs) and is not recommended for home/local setups.

Summary

  • For Speed: We reduced context. Try Q2_K quantization if you need more speed.
  • For Quality: Fine-tune Llama-3-8B using Unsloth, then quantize to Q4_K or Q2_K.