Logistic Regression / Neural Network from Scratch (PureMath Series)

A neural network built entirely from scratch using NumPy and autograd, with experiments on optimizers, loss functions, and class imbalance handling. This project is part of my puremath series, where I aim to deepen my understanding of machine learning from a mathematical perspective

The dataset used contains digital exposure parameters and a binary label indicating presence or absence of depression. The model predicts whether a person shows signs of depression based on these features.

This notebook is focused on my personal growth in AI/ML. The goal is not only performance optimization, but also:

• understanding the mathematical foundations of ML algorithms
• deriving and implementing key functions from scratch
• exploring how and why these methods were developed

I have attempted to explain the mathematical intuition behind most components used in this model and tried to prove all the mathematics I have used, what I havent understood is omitted from this notebook, and i have tried to give an alternate explaination to the same or I have just marked it was "im not sure".

Features of the Model

• Full neural network implementation from scratch (NumPy + autograd)
• Complete backpropagation implementation
• Multiple activation functions:
  • Sigmoid
  • ReLU
  • Tanh
• Optimization techniques:
  • SGD
  • Adam (RMSProp + Momentum / EWMA)
• Loss functions:
  • Crossentropy
  • Weighted Crossentropy
  • Focal Loss Crossentropy
• He/Kaiming initialization
• Custom evaluation metrics:
  • Precision
  • Recall
  • F1 Score
  • Threshold tuning

Dataset Features

• Highly imbalanced dataset (1139 non-depressed samples vs 31 depressed samples in X_train)
• Binary classification problem
• Severe class imbalance significantly affects performance metrics

Experiments Performed

• Baseline experimental model (logic validation)
• Weighted BCE + He initialization + Tanh + SGD
• Weighted BCE + He initialization + ReLU + SGD
• Focal Loss + He initialization + ReLU + SGD
• Focal Loss + He initialization + ReLU + Adam (best performance)

Final Results

• Best F1 Score: 0.13
• Best Threshold: 0.49

Comparison Table

Model Description	Loss Function	Optimizer	Initialization	F1 Score	Notes
Experimental Model	CrossEntropy	SGD	Random	-
Model Exp 1	Weighted CrossEntropy	SGD	He Init	0.07
Model Exp 2	Weighted CrossEntropy	SGD	He Init	-
Model Exp 3	Weighted CrossEntropy	SGD	He Init	0.01
Model Focal	Focal Loss	SGD	He Init	0.08
Model Focal (Adam)	Focal Loss	Adam	He Init	0.13	Best Model

Key Insights

• Optimizer Impact: Switching from SGD to Adam in the final experiment resulted in a significant performance boost, achieving the highest F1 score of 0.13.
• Loss Functions: Weighted CrossEntropy and Focal Loss were implemented to address data challenges, with Focal Loss showing the most promise when paired with the right optimizer.
• Initialization: Moving from Random to He Initialization helped stabilized training across all experimental and focal versions.

Key Learnings:

This notebook was not solely focused on maximizing F1 score. Instead, it was designed as a learning exercise to deeply understand the mathematical and algorithmic foundations of neural networks.

Through this project, I explored how different design choices affect learning behavior, especially under extreme class imbalance.

Future Improvements:

• Experiment with tree-based models (Random Forest / XGBoost)
• Hyperparameter tuning (not yet explored)
• Deeper neural network architectures
• Regularization techniques (Dropout, L2)
• Better handling of class imbalance (Feature engineering)

Final Note

This project is part of an ongoing learning journey in machine learning and mathematical AI foundations. The focus is on understanding, experimentation, andimprovement rather than achieving state-of-the-art performance.