Skip to content
xa-xPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Nano LLM

A tiny GPT-style language model for training on CPU, Apple Silicon, or NVIDIA GPU.

Features

  • 🚀 Multi-device support: MPS (Apple Silicon) | CUDA (NVIDIA) | CPU
  • 📦 30M parameters: Small enough for CPU training
  • 🔄 Scalable configs: micro → large (30M - 250M params)
  • 💬 Chat fine-tuning: Convert base model to chat assistant
  • ⚡ Auto-optimization: Batch size and settings tuned per device

Quick Start

# Setup
python3 -m venv venv
source venv/bin/activate
pip install torch numpy tiktoken datasets

# Check available devices
python3 device.py

# Train base model
python3 train_quick.py

# Chat with model
python3 chat.py "Hello!"

Device Support

🍎 Apple Silicon (M1/M2/M3/M4)

# Install PyTorch with MPS support
pip install torch

# Verify MPS is available
python3 -c "import torch; print(torch.backends.mps.is_available())"
# Should print: True

# Train (will auto-detect and use MPS)
python3 train_quick.py

🟢 NVIDIA GPU (CUDA)

# Install PyTorch with CUDA
pip install torch --index-url https://download.pytorch.org/whl/cu121

# Verify CUDA is available
python3 -c "import torch; print(torch.cuda.is_available())"
# Should print: True

# Train (will auto-detect and use CUDA)
python3 train_quick.py

💻 CPU (Fallback)

Works automatically if no GPU is detected.

Performance Comparison

Device Training Speed Inference Speed Batch Size
CPU (6 cores) ~5 steps/sec ~20 tok/s 4
MPS (M1) ~20-30 steps/sec ~100 tok/s 16
MPS (M2/M3) ~40-60 steps/sec ~200 tok/s 16
CUDA (RTX 3060) ~100 steps/sec ~500 tok/s 8
CUDA (RTX 3080) ~200 steps/sec ~800 tok/s 32
CUDA (RTX 4090) ~500 steps/sec ~2000 tok/s 64

Files

nano-llm/
├── config.py        # Model configs (micro → large)
├── model.py         # GPT architecture
├── device.py        # Device detection (MPS/CUDA/CPU)
├── train_quick.py   # Quick base training
├── train_chat.py    # Chat fine-tuning
├── chat.py          # Interactive chat
├── generate.py      # Text generation
└── out/
    ├── model.pt     # Base model checkpoint
    └── chat_model.pt # Chat model checkpoint

Training

Base Model (Story Generation)

python3 train_quick.py
  • 500 steps, batch size auto-tuned for your device
  • Outputs: out/model.pt

Chat Model (Conversational AI)

python3 train_chat.py
  • 300 steps, fine-tunes base model
  • Outputs: out/chat_model.pt

Inference

Interactive Chat

python3 chat.py

Single Prompt

python3 chat.py "Tell me a joke"
python3 generate.py out/model.pt "Once upon a time"

Model Configs

Name Params Layers Heads Embed Use Case
micro ~30M 6 6 384 Testing, CPU
tiny ~50M 8 8 512 Quick experiments
small ~80M 10 10 640 Decent quality
medium ~125M 12 12 768 Cloud training
large ~250M 16 16 1024 Production

Cloud Training

For larger models, use free cloud GPUs:

Google Colab (Free T4 GPU)

!git clone <your-repo>
!pip install torch tiktoken datasets
!python device.py  # Should show CUDA available
!python train_quick.py

Kaggle (Free P100 GPU)

  • 30 hours/week free GPU
  • Enable GPU in Notebook settings

Device Detection Details

The device.py module automatically:

  1. Checks for Apple Silicon MPS
  2. Falls back to NVIDIA CUDA
  3. Falls back to CPU
from device import get_device, get_device_info, optimize_for_device

# Print all device info
print(get_device_info())

# Get best device
device = get_device()

# Get optimal settings
opts = optimize_for_device(device)
print(f"Batch size: {opts['batch_size']}")

Requirements

torch>=2.0
numpy
tiktoken
datasets

License

MIT

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages