Teach Your Computer to Write: Code and Exercises

Companion code for the book Teach Your Computer to Write: Build and Train an LLM from Zero by Truong (Jack) Luu, a free, hands-on book written for business students. The book builds a small GPT-style language model from scratch in Python and PyTorch, in 17 chapters and 3 modules, and trains it on an ordinary computer with no graphics card. It was first published as Building an LLM from Zero.

• Read the book online: https://jackluu.io/book/
• Download the PDF: https://jackluu.io/files/teach-your-computer-to-write.pdf
• Download the EPUB: https://jackluu.io/files/teach-your-computer-to-write.epub
• Teach a course with it: https://jackluu.io/book/teach/

This repository holds the runnable code for every chapter, the short "Try It" examples, the exercises and assignments, and a notebook per chapter. The book explains each file line by line.

Run it in your browser

Every chapter has a notebook in notebooks/. They need nothing installed: open one in Colab and the first cell fetches the code, installs PyTorch and downloads the text.

• Chapter 1: notebook | open in Colab
• Chapter 2: notebook | open in Colab
• Chapter 3: notebook | open in Colab
• Chapter 4: notebook | open in Colab
• Chapter 5: notebook | open in Colab
• Chapter 6: notebook | open in Colab
• Chapter 7: notebook | open in Colab
• Chapter 8: notebook | open in Colab
• Chapter 9: notebook | open in Colab
• Chapter 10: notebook | open in Colab
• Chapter 11: notebook | open in Colab
• Chapter 12: notebook | open in Colab
• Chapter 13: notebook | open in Colab
• Chapter 14: notebook | open in Colab
• Chapter 15: notebook | open in Colab
• Chapter 16: notebook | open in Colab
• Chapter 17: notebook | open in Colab

Setup

Chapter 1 walks through every step. In short, with Python 3.11 or newer:

git clone https://github.com/jackluucoding/build-llm-from-zero.git
cd build-llm-from-zero
python -m venv .venv
# Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cpu
pip install numpy requests matplotlib
python src/utils/download_data.py
python src/ch00_setup_check.py

Run every file from the repository folder, for example python src/ch03_tokenizer.py. File numbers are one lower than chapter numbers because the setup check is ch00.

Chapters and files

Ch	Chapter	Main file	Try It examples
1	Environment Setup	src/utils/download_data.py, src/ch00_setup_check.py	-
2	What Is an LLM?	(concepts only)	src/examples/ch02_next_char.py
3	Tensors and PyTorch	src/ch02_tensors.py	-
4	Tokenization	src/ch03_tokenizer.py	src/examples/ch04_tokenizer_demo.py
5	Embeddings	src/ch04_embeddings.py	-
6	Self-Attention (Single Head)	src/ch05_self_attention.py	-
7	Multi-Head Attention	src/ch06_multihead_attention.py	src/examples/ch07_head_budget.py, src/examples/ch07_heads_differ.py
8	Feed-Forward and Norms	src/ch07_feedforward.py	-
9	The Transformer Block	src/ch08_transformer_block.py	-
10	The Full GPT Architecture	src/ch09_gpt_model.py	-
11	Causal Language Modeling	src/ch10_causal_lm.py	src/examples/ch11_loss_demo.py
12	Dataset and DataLoader	src/ch11_dataloader.py	src/examples/ch12_batch_speed.py, src/examples/ch12_batching_demo.py
13	The Training Loop	src/ch12_train.py	src/examples/ch13_autograd_demo.py, src/examples/ch13_learning_rate.py, src/examples/ch13_loss_meaning.py
14	Checkpointing	src/ch13_checkpoint.py	src/examples/ch14_inspect_checkpoint.py
15	Greedy and Sampling	src/ch14_generate_greedy.py	src/examples/ch15_different_prompts.py
16	Temperature and Top-k	src/ch15_generate_sampling.py	src/examples/ch16_explore_temp.py, src/examples/ch16_temperature.py
17	Putting It All Together	src/ch16_full_pipeline.py	src/examples/ch17_deploy.py

Exercises

Each chapter's hands-on tasks, review questions and assignment. Every assignment takes about 15 minutes: run a script, change one value, and explain what changed.

• Chapter 1
• Chapter 2
• Chapter 3
• Chapter 4
• Chapter 5
• Chapter 6
• Chapter 7
• Chapter 8
• Chapter 9
• Chapter 10
• Chapter 11
• Chapter 12
• Chapter 13
• Chapter 14
• Chapter 15
• Chapter 16
• Chapter 17

Data

src/utils/download_data.py downloads the Tiny Shakespeare text from Andrej Karpathy's char-rnn project (https://github.com/karpathy/char-rnn) into src/data/.

How to cite

Use the "Cite this repository" button on this page, or see How to Cite on the book's site.

License

• Code (src/) and notebooks (notebooks/): MIT, see LICENSE.
• Exercises (exercises/): CC BY-NC 4.0, so that instructors may adapt them for their own courses.
• The book's text and lecture decks, at https://jackluu.io/book/: CC BY-NC-ND 4.0. LICENSE-BOOK sets out all three.

Author

Truong (Jack) Luu, Ph.D., Assistant Professor of Information Systems and Analytics, McCoy College of Business, Texas State University. https://jackluu.io