nanoLM 📋

nanoLM is a repository documenting my journey building Language Models from scratch, following Andrej Karpathy's legendary "Neural Networks: Zero to Hero" series.

The goal is to demystify deep learning and NLP by starting with the simplest possible components and iteratively building up to a full-scale GPT implementation. The current focus is on character-level language modeling using the names.txt dataset.

---

🚀 Progress & Roadmap

Model Architecture	Status	Key Concepts
Bigram Language Model	✅ Done	Probability tables, NLL Loss, One-hot encoding, Basic PyTorch
Multi-Layer Perceptron (MLP)	✅ Done	Embeddings, Hidden Layers, Tanh activation, Context Window (Bengio et al. 2003)
Batch Norm & Internals	⏳ In Progress	Weight initialization, Batch Normalization, Diagnostic visualizations
Manual Backpropagation	🔮 Planned	Calculating gradients by hand, understanding the "under the hood" mechanics
Wavenet / CNN	🔮 Planned	Hierarchical interactions, dilated convolutions
Transformer (GPT)	🔮 Planned	Self-attention, Positional encodings, Scaled dot-product attention

---

📂 Project Modules

1. Bigram Language Model (bigramv1.ipynb)

The Foundation. We started by building a purely statistical model that predicts the next character based solely on the preceding character.

• Approach: Count-based probability lookup table vs. Single-layer Neural Network.
• Metric: Negative Log Likelihood (NLL).
• Result: The model learns local patterns (e.g., 'q' is often followed by 'u') but fails to generate coherent names because it lacks context.

2. Multi-Layer Perceptron (mlp.ipynb)

Adding Brains & Context. Moving beyond simple statistics, this module implements the architecture proposed by Bengio et al. (2003).

• Architecture:
  • Embedding Layer: Maps characters to a dense vector space (learning similarity between characters).
  • Context Window: Takes a block of previous characters (block_size=3) to predict the next one.
  • Hidden Layer: Uses Tanh activation for non-linearity.
• Improvement: The model now understands structure better (vowels follow consonants, endings look realistic).

---

🧪 Model Outputs & Comparison

Here is how the generated names evolved as the architecture became more complex:

MLP Generation Samples

With embeddings and a context window, the model starts hallucinating plausible names:

Harlin Mili Taty Ham

(Note: While not perfect, these are significantly more "name-like" compared to the Bigram model's random character thingy!)

---

🛠️ Tech Stack

• Python 3
• PyTorch: For building and training neural networks.
• Matplotlib: For visualizing loss curves and character embeddings.

🏃 How to Run

1. Clone the repository:

   git clone [https://github.com/yourusername/nanoLM.git](https://github.com/yourusername/nanoLM.git)
   cd nanoLM

2. Install dependencies:

   pip install torch matplotlib

3. Open the notebooks (Jupyter/Colab) and run the cells to see the training process in real-time.

---

Acknowledgements

Huge thanks to Andrej Karpathy for the incredible Zero to Hero series. This repo is my attempt to code along and truly understand the magic behind LLMs.