slimGPT

124M-parameter GPT trained from scratch — clean, file-by-file, readable.

---

slimGPT trains GPT-2 small (12 layers · 12 heads · 768-dim · 124M params) on OpenWebText.
Same architecture and recipe, reorganised so every concern lives in its own file and nothing is hidden.

tokens → [embed + pos] → [12× Block(Attention + MLP + LayerNorm)] → [LayerNorm] → logits

slimgpt/
├── config.py          GPTConfig — architecture hyper-parameters
├── layers.py          LayerNorm, CausalSelfAttention, MLP, Block
├── model.py           GPT — forward, from_pretrained, generate, configure_optimizers
├── data.py            DataLoader — batches from tokenised .bin files
├── lr.py              cosine LR schedule with linear warmup
└── train_config.py    TrainConfig — training hyper-parameters + CLI

---

Results

Training run: 5 000 iterations on ~5B tokens of OpenWebText, single NVIDIA L4 (23 GB VRAM).

Checkpoint	Train loss	Val loss
1 000	4.1839	4.1632
2 000	3.6316	3.6346
3 000	3.4578	3.4535
4 000	3.3594	3.3680
5 000	3.2934	3.3079

Perplexity (best val): 27.3 · GPT-2 small fully trained ≈ 22.4

Val loss tracked train loss closely throughout — generalisation gap at 5k iters: +0.0145 (minimal overfitting).

---

Install

git clone https://github.com/samueljayasingh/slimGPT
cd slimGPT
pip install -r requirements.txt
# install PyTorch for your CUDA version: https://pytorch.org

---

Quickstart

Verify the pipeline in a few minutes on CPU with tiny-shakespeare:

python data/shakespeare_char/prepare.py
python train.py --config configs/shakespeare_char.py --device=cpu --compile=False
python sample.py --out_dir=out-shakespeare-char --device=cpu

---

Train GPT-124M

The full OpenWebText (~54 GB raw) doesn't fit on a 60 GB disk. prepare.py streams the corpus and writes only a capped token budget directly to .bin — raw text is never stored.

# ~5B tokens, ~10 GB (default)
python data/openwebtext/prepare.py

# smaller, ~6 GB
python data/openwebtext/prepare.py --train_tokens 3e9

Single GPU:

python train.py --config configs/gpt124m.py

Multi-GPU (single node):

torchrun --standalone --nproc_per_node=8 train.py --config configs/gpt124m.py

gradient_accumulation_steps is divided across ranks automatically — effective batch stays constant.

Sample from a trained checkpoint:

python sample.py --out_dir=out-gpt124m --start="The meaning of life is" --num_samples=5

---

Export

The training checkpoint (out-gpt124m/ckpt.pt) carries optimizer state. export.py converts it to portable formats:

# HuggingFace (model + tokenizer + model card)
python export.py --out_dir=out-gpt124m --format=hf --dest=export/slimGPT

# push directly to the Hub
python export.py --out_dir=out-gpt124m --format=hf \
    --push_to_hub --repo_id=<your-username>/slimGPT

# inference-only .pt (no optimizer state, ~half the size)
python export.py --out_dir=out-gpt124m --format=pt --dest=export/slimGPT-slim.pt

# single safetensors file
python export.py --out_dir=out-gpt124m --format=safetensors --dest=export/slimGPT.safetensors

Load from HuggingFace:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("samueljayasingh/slimGPT")
model = AutoModelForCausalLM.from_pretrained("samueljayasingh/slimGPT")

ids = tok("The meaning of life is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=100)[0]))

---

Config overrides

Any TrainConfig field can be set inline:

python train.py --config configs/gpt124m.py \
    --max_iters=10000 \
    --learning_rate=3e-4 \
    --wandb_log=True

Resume an interrupted run:

python train.py --config configs/gpt124m.py --init_from=resume

Fine-tune from OpenAI's released weights:

python train.py --config configs/gpt124m.py --init_from=gpt2

Load pretrained weights directly in Python:

from slimgpt import GPT
model = GPT.from_pretrained("gpt2")  # also: gpt2-medium, gpt2-large, gpt2-xl

---

Training hardware

Component	Spec
GPU	NVIDIA L4 — 23 GB VRAM
CPU	Intel Xeon @ 2.20 GHz (4 vCPUs)
RAM	15 GiB
OS	Debian GNU/Linux 12 (Bookworm)
Driver	NVIDIA 550.54.15
Tokens/iter	491 520
Throughput	~40 K tokens/s

---

Training Runtime

• Hardware: NVIDIA L4 (24 GB VRAM), 4 vCPUs, 16 GB RAM
• Training iterations: 5,000
• Total training time: ~18 hours
• Average time per iteration: ~13 seconds

---

Notes

• Flash Attention — used automatically on PyTorch ≥ 2.0; falls back to masked-softmax on older builds.
• torch.compile — on by default. Requires PyTorch 2.0+ and a recent GPU. Disable with --compile=False.
• Precision — bfloat16 by default on Ampere+ GPUs. Use --dtype=float16 (with GradScaler) on older cards, or --dtype=float32 on CPU.
• Effective batch = batch_size × block_size × grad_accum × world_size. The GPT-124M preset targets ~0.5M tokens/iter.

---

Credits

Inspired by Andrej Karpathy's "Let's reproduce GPT-2 (124M)" tutorial. Special thanks to Andrej Karpathy for making modern LLM training and implementation accessible through open educational content.

---

MIT License · built by Samuel Jayasingh