🧠 Neural Network from Scratch

This project implements a simple neural network built entirely from scratch using NumPy — without relying on deep learning frameworks like TensorFlow or PyTorch.

The main goal of this project was to understand and visualize how neural networks actually work under the hood — including forward propagation, backpropagation, and gradient descent — while keeping the code as readable and transparent as possible.

⚠️ Note: During development, I intentionally did not focus on advanced software engineering principles such as SOLID, design patterns, or heavy abstraction layers.
The focus here is clarity, mathematical transparency, and educational value, not production-level architecture.

---

🚀 Features

• Implemented ReLU and Softmax activation functions
• Cross-Entropy Loss with stable numerical handling
• Backpropagation implemented manually
• Weights & Biases (wandb) integration for optional experiment tracking
• Tkinter interface for interactive digit drawing and model testing
• Dockerized environment for reproducibility

---

🧩 Project Structure

neuralnet-from-scratch/
├── src/
│   ├── interface.py        # Tkinter GUI for digit drawing
│   ├── neural_network.py   # Core neural network logic (forward/backward)
│   └── load_mnist.py       # Load and split MNIST dataset
├── weights/                # Saved trained model weights
├── main.py                 # Entry point to run the app
├── requirements.txt
├── dockerfile.yaml
├── docker-compose.yaml
└── README.md

---

🚀 Running

1️⃣ Clone the Repository

First, create your project folder and clone this repository:

# Create a working directory
mkdir neuralnet-from-scratch
cd neuralnet-from-scratch

# Clone the repository
git clone https://github.com/MikolajRusin/NeuralNetwork-From-Scratch.git .

⚙️ Optional: Weights & Biases Setup

If you want to enable experiment tracking with Weights & Biases (wandb), follow these steps:

1️⃣ Go to the official website and sign in:
👉 https://wandb.ai/

2️⃣ Create a new project, for example:
NeuralNetwork-from-scratch

3️⃣ Copy your personal API key:
If you don't see the API Key after created new project, then:
(Account → API Key → Copy)

4️⃣ Open your .env file in the project root and update it as follows:*

• Set USE_WANDB=True
• Paste your copied API key into WANDB_API_KEY

USE_WANDB=True               # Enable wandb logging
WANDB_API_KEY=your_api_key_here
WANDB_PROJECT=NeuralNetwork-from-scratch

⚠️ Note: If you don’t want to use Weights & Biases, simply set:
USE_WANDB=False

🧩 Model Hyperparameters

You can also customize the model architecture directly in your .env file:

# Model's hyperparameters
INPUT_LAYER=784
HIDDEN_LAYERS=100
OUTPUT_LAYER=10

• INPUT_LAYER defines the number of input neurons (for MNIST this is always 28×28 = 784).
• HIDDEN_LAYERS sets the number of neurons in the hidden layer(s).
  You can define multiple layers using commas, e.g. HIDDEN_LAYERS=128,64.
• OUTPUT_LAYER defines the number of output neurons (for MNIST → 10, digits 0–9).

💡 Important: The pre-trained model weights (trained_weights.npy) included in this project were trained using one hidden layer with 100 neurons. If you modify the architecture in your .env, these weights will no longer match the new layer structure.

💻 Running Locally (Without Docker)

# 1️⃣ Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate       # On Windows use: .venv\Scripts\activate

# 2️⃣ Install dependencies
pip install --upgrade pip
pip install -r requirements.txt

# 4️⃣ Run the application
python main.py

🐳 Running with Docker

# 🐧 On Linux:
xhost +local:docker
docker-compose up --build

🖼️ Tkinter Interface Overview

Once you run the application successfully, a Tkinter window should appear.
This interface allows you to interact with the neural network in real time — drawing digits, training the model, and adjusting key parameters.

Below is a description of each main element visible in the GUI:

🧩 Interface Elements

• 🧠 Load Weights – Opens a file dialog that lets you select a pre-trained model file (.pkl or .npy) from the weights/ folder.
  After loading, the neural network is ready to make predictions immediately.

  If you want to quickly test how the model performs, you can load the pre-trained weights file
  trained_weights.npy, which is already included in the project.
  These weights were trained for 300 epochs on the MNIST dataset.
  The resulting performance metrics and training progress can be later observed on the provided plots in the results section of the project.

• 📈 Train Model – Starts the training process on the MNIST dataset.
  During training, progress is displayed in the terminal or console window, showing the loss and accuracy value for each epoch.

• ⚙️ Learning Rate – A text field or input box where you can set the learning rate (e.g., 0.001, 0.01, 0.1).
  This parameter determines how strongly the model updates its weights after each batch.

• 🔁 Epochs – Defines the number of training epochs (complete passes through the dataset).
  A higher number may improve accuracy but increases training time.

💡 Tip: You can train a model once, the weights will be saved into the weights/ directory, and later you can load them again using the Load Weights button — without retraining every time.

---

📊 Training Results & Plots

After training the model for 300 epochs on the MNIST dataset,
the following results were obtained:

• Training loss: ~0.59
• Validation loss: ~0.76
• Training accuracy: ~85%
• Training accuracy: ~85%
• Validation accuracy: ~82%

The plots below show how the loss decreased and accuracy improved over time:

Training loss and accuracy progression during training. (300 epochs)

💡 You can reproduce these results by retraining the model using the Train Model button,
or by running the training script directly (main.py).

---

🚀 Possible Improvements

As mentioned earlier, the main goal of this project was to build a simple and transparent neural network from scratch — focusing on clarity and understanding rather than advanced engineering or optimization.
That said, there are many ways in which this model could be significantly improved in both performance and efficiency.

⚙️ 1️⃣ Activation Function

In this implementation, the hidden layers use the ReLU (Rectified Linear Unit) activation function:

$$ ReLU(x)=\max(0, x) $$

While ReLU is simple and effective, it is not differentiable at 0, which can lead to dead neurons during training.
A more robust alternative would be to use Leaky ReLU, which allows a small gradient when the unit is inactive:

$$ LeakyReLU(x) = \begin{cases} x, & \text{if } x > 0 \\ \alpha x, & \text{if } x \leq 0 \end{cases} $$

This small change often improves gradient flow and model stability.

---

⚙️ 2️⃣ Optimizer

Currently, the model updates weights using full-batch gradient descent,
which adjusts all weights after processing the entire dataset once per epoch:

$$ w = w - \eta \frac{\partial L}{\partial w} $$

This is simple but computationally inefficient for larger datasets.

Possible improvements:

• Implement Mini-Batch Gradient Descent (SGD) — updates weights using small batches of data.
• Implement Momentum to accelerate convergence.
• Implement Adam or AdamW optimizers, which adapt the learning rate dynamically for each parameter.

---

⚙️ 3️⃣ Weight Initialization

Weights are currently initialized with a standard normal distribution.
For deeper networks, consider:

• He Initialization (for ReLU / LeakyReLU activations)
• Xavier Initialization (for tanh or sigmoid activations)

These methods help prevent vanishing or exploding gradients.

---

⚙️ 4️⃣ Regularization and Generalization

To improve model generalization and avoid overfitting:

• Add Dropout layers between hidden layers.
• Implement L2 regularization (weight decay) in the loss function.

The L2 regularization term penalizes large weights by adding their squared magnitude to the total loss:

$$ L_{\text{total}} = L_{\text{data}} + \lambda \sum w^2 $$

Where:

• L_total — total loss function,
• L_data — the original loss (e.g., cross-entropy),
• λ — the regularization coefficient (controls the penalty strength),
• Σ w² — the sum of squared weights across all layers.

This encourages the network to keep weights small and prevents it from relying too heavily on specific neurons —
resulting in a smoother, more generalizable model.

💡 In practice, you can implement it in your training code like this:

loss = cross_entropy(y_pred, y_true)
l2_term = sum(np.sum(w ** 2) for w in self.weights)
loss += lambda_ * l2_term

---

⚙️ 5️⃣ Additional Enhancements

• Support dynamic architectures directly via .env (e.g. multiple hidden layers [128, 64, 32]).
• Introduce early stopping to stop training when validation loss stops improving.
• Implement visualization of predictions directly in the Tkinter interface (e.g., probability bars or confusion matrix).

---

💡 These changes would transform this educational neural network into a more advanced and flexible model —
bridging the gap between a minimal NumPy implementation and modern deep learning frameworks like PyTorch or TensorFlow.