GoLLuM

Go Large Language Model - Building an LLM from Scratch in Go

Project Overview

GoLLuM is an experimental project for building a Large Language Model from scratch in Go. This project translates the Python examples from the book "Build a Large Language Model (From Scratch)" by Sebastian Raschka into idiomatic Go, utilizing open-source machine learning packages from the Gonum and Gorgonia ecosystems.

The goal is to provide a Go-native implementation of the concepts and architectures presented in the book, making LLM development accessible to the Go community while demonstrating the capabilities of Go's ML tooling.

Why Go?

• Simplicity and Performance: Go's straightforward syntax and excellent performance characteristics make it ideal for educational implementations
• Concurrency: Native goroutines and channels provide powerful primitives for parallel data processing
• Deployment: Single binary compilation and cross-platform support simplify model deployment
• Growing ML Ecosystem: Projects like Gorgonia, Gonum, and others are making ML in Go increasingly viable

Dependencies

This project leverages several key Go ML libraries:

• Gorgonia - Deep learning framework with automatic differentiation
• Gorgonia Tensor - N-dimensional array library (similar to NumPy)
• Gonum - Numerical computing libraries
• tiktoken-go - BPE tokenizer implementation (OpenAI's tokenizer)

Project Structure

The project is organized by book chapters, with each chapter containing:

• cmd/ - Executable examples demonstrating concepts
• pkg/ - Reusable library code for core functionality

GoLLuM/
├── chapter2/              # Working with Text Data
│   ├── cmd/
│   │   ├── bpe/               # Byte Pair Encoding example
│   │   ├── tokenize/          # Basic tokenization
│   │   ├── tokenizer/         # Advanced tokenizer usage
│   │   ├── sampling/          # Data sampling strategies
│   │   ├── batchsampling/     # Batch sampling with DataLoader
│   │   └── embedding/         # Token embedding examples
│   └── pkg/
│       ├── tokenizer/         # Custom tokenization utilities
│       ├── dataset/           # Dataset and DataLoader implementations
│       └── embedding/         # Embedding layer implementation
├── chapter3/              # Coding Attention Mechanisms
│   ├── cmd/
│   │   ├── simpleattention/   # Scalar and vector attention walkthroughs
│   │   ├── selfattention/     # Matrix self-attention (single query)
│   │   ├── selfattention2/    # Self-attention using reusable SelfAttention type
│   │   ├── trainableweights/  # Attention with learned Q/K/V projections
│   │   ├── causalattention/   # Causal masking (simple and robust -inf versions)
│   │   ├── causalattention2/  # Batched causal attention using CausalAttention type
│   │   ├── multihead/         # Simple wrapper multi-head attention (multiple CausalAttention heads)
│   │   └── multiheadsplitweights/ # Multi-head attention with shared projections and head splits
│   └── pkg/
│       └── attention/         # Self-, causal-, and multi-head attention building blocks
├── go.mod
└── README.md

Implementation Progress

✅ Chapter 2: Working with Text Data

Status: Complete

Chapter 2 covers the fundamental data processing pipeline for LLMs:

• Tokenization: Converting raw text into tokens

  • Simple regex-based tokenization with punctuation handling
  • BPE (Byte Pair Encoding) using tiktoken
  • Vocabulary building and management
  • Encoding/decoding with proper punctuation spacing

• Embeddings: Converting tokens to vector representations

  • Token embedding layer with configurable dimensions
  • Seeded weight initialization for reproducibility
  • Integration with Gorgonia's automatic differentiation

• Data Sampling: Creating training samples from text

  • Sliding window approach with configurable stride
  • Input/target pair generation for next-token prediction
  • Efficient batching with DataLoader
  • Shuffle, drop-last, and batch size controls

Key Components:

• GPTDataset: Generates input/target pairs from tokenized text
• DataLoader: Batches samples with shuffling and iteration support
• Embedding: Token embedding layer compatible with Gorgonia
• Tokenizer: Text preprocessing and tokenization utilities

✅ Chapter 3: Coding Attention Mechanisms

Status: Complete

Chapter 3 implements a series of attention mechanisms, step by step:

• Scalar and vector attention:

  • Manual dot-product attention with a single query
  • Score normalization vs. proper softmax
  • Numerical checks against the book’s PyTorch examples

• Self-attention:

  • Matrix-based self-attention (inputs @ inputs.T)
  • Reusable SelfAttention type in chapter3/pkg/attention
  • Row-wise softmax implementation (RowWiseSoftmax) for correct per-token normalization

• Trainable attention weights:

  • Learned W_query, W_key, and W_value projections
  • Seeded uniform initializers (SeededUniform) to mirror torch.rand with deterministic output

• Causal attention:

  • Lower-triangular masks applied in weight space (simple masking)
  • Robust masking in score space using -Inf and softmax (true causal attention)
  • CausalAttention type for batched inputs with optional dropout

• Multi-head attention:

  • Wrapper-style multi-head attention using multiple CausalAttention heads
  • Full MultiHeadAttention implementation that:
    • Uses shared W_query, W_key, W_value projections
    • Splits heads via reshaping/slicing
    • Applies causal masks and row-wise softmax per head
    • Recombines heads and applies an output projection (W_out_proj)

Key types and components:

• SelfAttention, CausalAttention, MultiHeadAttention in chapter3/pkg/attention
• Example executables under chapter3/cmd/ demonstrating each concept in isolation

✅ Chapter 4: Implementing a GPT Model

Status: Complete

Chapter 4 begins assembling the components from earlier chapters into a GPT-style model:

• Dummy GPT model:

  • DummyGPTModel with token and positional embeddings
  • Dropout on embeddings
  • Stack of Transformer blocks (DummyTransformerBlock)
  • Final LayerNorm and output projection to vocabulary logits

• Layer normalization:

  • DummyLayerNorm implementing per-token LayerNorm over the embedding dimension
  • Verified with standalone normalization example in chapter4/cmd/normalizing

• Feed-forward network & GELU:

  • GELU activation (tanh-based approximation)
  • FeedForward MLP: emb_dim → 4*emb_dim → emb_dim

• Transformer block:

  • DummyTransformerBlock combining:
    • Multi-head causal self-attention (MultiHeadAttention from Chapter 3)
    • Feed-forward MLP
    • Pre-LayerNorm and residual connections for both attention and MLP paths
    • Per-block dropout on residual branches

• Text generation:

  • GenerateTextSimple helper: greedy decoding with a context window
  • chapter4/cmd/generatetextsimple example:
    • Uses tiktoken-go for GPT-2 tokenization
    • Runs untrained DummyGPTModel forward
    • Demonstrates end-to-end text generation before training

📋 Chapter 5: Pretraining on Unlabeled Data (Planned)

Will implement:

• Training loop
• Loss calculation
• Gradient descent optimization
• Model checkpointing
• Training metrics and monitoring

📋 Chapter 6: Finetuning for Classification (Planned)

Will implement:

• Classification head
• Transfer learning
• Task-specific finetuning
• Evaluation metrics

📋 Chapter 7: Finetuning to Follow Instructions (Planned)

Will implement:

• Instruction formatting
• Supervised finetuning
• Response generation
• Model evaluation

Getting Started

Prerequisites

• Go 1.23.4 or later
• Basic understanding of machine learning concepts
• Familiarity with the original book is helpful but not required

Installation

git clone https://github.com/yourusername/GoLLuM.git
cd GoLLuM
go mod download

Running Examples

Each chapter contains executable examples in the cmd/ directory:

# Chapter 2 Examples
cd chapter2

# Run tokenization example
go run cmd/tokenize/main.go

# Run embedding example
go run cmd/embedding/main.go

# Run batch sampling example
go run cmd/batchsampling/main.go

Key Differences from the Python Implementation

1. Type Safety: Go's static typing catches errors at compile time that might be runtime errors in Python
2. Explicit Error Handling: All errors are explicitly handled rather than relying on exceptions
3. Tensor Operations: Using Gorgonia's tensor library instead of PyTorch
4. Memory Management: More explicit control over memory allocation and copying
5. Concurrency: Potential for parallel processing using goroutines (where applicable)

Learning Resources

• Original Book: Build a Large Language Model (From Scratch) by Sebastian Raschka
• Gorgonia Documentation: gorgonia.org
• Go ML Resources: awesome-go-ml

Development Notes

Design Decisions

• Package Structure: Reusable components are in pkg/, examples in cmd/
• Interface-Driven: Using Go interfaces (e.g., Dataset) for flexibility
• Explicit Over Implicit: Preferring clarity and explicit behavior over "magic"
• Faithful Translation: Staying close to the book's Python examples while being idiomatic Go

Testing

Run tests with:

go test ./...

Contributing

This is a learning project and contributions are welcome! If you find bugs, have suggestions, or want to implement additional chapters:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Please maintain the code style and include comments explaining ML concepts where appropriate.

License

[To be determined - follow the same licensing as the original book's code samples]

Acknowledgments

• Sebastian Raschka for the excellent book and clear explanations
• The Gorgonia and Gonum teams for building Go's ML ecosystem
• The Go community for making systems programming accessible and enjoyable

Project Status

🚧 Active Development - Chapters 2 and 3 implemented; Chapter 4 underway

This project is in early stages. Chapters 2 through 4 are complete, with a working GPT-style scaffold (embeddings, attention blocks, LayerNorm, feed-forward, and simple text generation) demonstrating the feasibility of building and running a GPT-like model in Go. Subsequent chapters will focus on pretraining, finetuning, and instruction tuning to turn this scaffold into a fully trained GPT-style language model.

---

Note: This is an educational project. For production LLM applications, consider using established frameworks and pre-trained models. This project is ideal for:

• Learning how LLMs work under the hood
• Understanding Go's ML capabilities
• Experimenting with custom architectures
• Educational purposes and research