CTM-Enhanced Spiking Neural Network for MNIST

A biologically-inspired spiking neural network that achieves 98.10% accuracy on MNIST while maintaining biological realism through excitatory/inhibitory balance and temporal pattern analysis.

🧠 Key Features

• Continuous Thought Machine (CTM) algorithms for iterative reasoning
• Biologically-realistic excitatory/inhibitory balance (80/20 ratio)
• Temporal pattern analysis beyond simple rate coding
• Multi-head attention with temporal biases
• State-of-the-art performance (98.10% on MNIST)
• Energy-efficient sparse spiking computation

🚀 Quick Start

Installation

git clone https://github.com/deanhorak/snn.git
cd snn
pip install -r requirements.txt

Training

python train_mnist.py

Key Results

• Final Accuracy: 98.10% on MNIST test set
• Training Time: ~2 hours on GPU
• Spike Sparsity: Only ~5% of neurons active per input
• Biological Realism: Maintained throughout training

📊 Performance Comparison

Method	Accuracy (%)	E/I Balance	Temporal Patterns	Energy Efficiency
Traditional CNN	99.7%	No	No	Low
DIET-SNN	99.61%	No	No	Medium
CTM-SNN (Ours)	98.10%	Yes	Yes	High

🏗️ Architecture Overview

Input (784) → SNN Layer 1 (512) → CTM Processing → SNN Layer 2 (10) → Output
                ↓                      ↓                    ↓
        E/I Balance (80/20)    Iterative Reasoning    Classification

CTM Processing Steps:

1. Multi-head attention with temporal biases
2. Population synchronization analysis
3. Rate-coded integration with memory
4. Iterative refinement (3 thinking steps)

📁 Project Structure

snn_mnist/
├── README.md                 # This file
├── requirements.txt          # Dependencies
├── train_mnist.py           # Main training script
├── snn_model.py             # CTM-SNN architecture
├── utils.py                 # Helper functions
├── config.py                # Configuration settings
├── results/                 # Training results and logs
├── models/                  # Saved model checkpoints
└── notebooks/               # Jupyter notebooks for analysis
    ├── temporal_analysis.ipynb
    └── visualization.ipynb

🔬 Key Innovations

1. Continuous Thought Machine Integration

# CTM processing with iterative reasoning
for step in range(num_thinking_steps):
    attention_out = self.multi_head_attention(spikes, step)
    sync_out = self.synchronization_analysis(spikes)
    rate_out = self.rate_coding(spikes)
    
    thought_state = self.ctm_step(attention_out, sync_out, rate_out, thought_state)

2. Temporal Pattern Analysis

• Timing preferences: When neurons prefer to fire
• Burst patterns: Rapid firing sequences
• Temporal variance: Spread of spike timing
• Early/late bias: Temporal preference ratios

3. Biologically-Realistic E/I Balance

• 80% excitatory neurons (threshold = 1.0)
• 20% inhibitory neurons (threshold = 0.8)
• Lateral inhibition for winner-take-all dynamics

📈 Training Details

Hyperparameters

• Time steps: 16 per input
• Thinking steps: 3 CTM iterations
• Learning rate: 1e-3 with cosine annealing
• Batch size: 128
• Epochs: 35 with early stopping

Advanced Techniques

• Stochastic Weight Averaging (SWA): From epoch 11
• Mixup augmentation: After epoch 5
• Test-time augmentation (TTA): Multiple views
• Surrogate gradients: For spike function training

🧪 Experimental Results

Training Progression

• Epoch 1: 92.44% test accuracy
• Epoch 13: 97.13% (entering high-performance range)
• Epoch 35: 98.10% (final performance)

Temporal Pattern Evolution

• Timing: 0.196 → 0.230 (learned optimal firing times)
• Variance: 0.036 → 0.043 (increased temporal diversity)
• Bursts: 0.364 → 0.447 (emergence of burst coding)

Ablation Study

Component Removed	Accuracy Drop
E/I Balance	-2.5%
CTM Thinking	-3.9%
Attention Heads	-1.3%
SWA	-0.4%

🔧 Usage Examples

Basic Training

from snn_model import CTMSNN
from train_mnist import train_model

# Create model
model = CTMSNN(
    input_size=784,
    hidden_size=512,
    output_size=10,
    time_steps=16,
    thinking_steps=3
)

# Train model
train_model(model, epochs=35)

Temporal Analysis

# Analyze temporal patterns
temporal_patterns = model.extract_temporal_patterns(spike_data)
timing = temporal_patterns[:, :512]  # Timing preferences
variance = temporal_patterns[:, 512:1024]  # Temporal variance
bursts = temporal_patterns[:, 1024:1536]  # Burst patterns

🚀 Future Directions

• Scaling to CIFAR-10/ImageNet: Larger, more complex datasets
• Neuromorphic hardware: Intel Loihi implementation
• Multi-modal processing: Audio-visual fusion
• Continual learning: Online adaptation capabilities

📚 Citation

If you use this code in your research, please cite:

@article{horak2024ctmsnn,
  title={Beyond Rate Coding: Building Competitive Spiking Neural Networks with CTM Algorithms},
  author={Horak, Dean S.},
  journal={arXiv preprint},
  year={2024}
}

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

Contributions welcome! Please read CONTRIBUTING.md for guidelines.

📞 Contact

• Author: Dean S. Horak
• Email: dean.horak@example.com
• GitHub: @deanhorak

---

This project demonstrates that biological realism and computational performance are not mutually exclusive in artificial neural networks.