The Singularity AI (Singularity-7B)

Bespoke C++/CUDA 7 Billion Parameter Large Language Model Training & Inference Engine
Engineered from scratch without PyTorch or standard ML frameworks, featuring a 128k token context window via Tiled FlashAttention, custom fused CUDA kernels, arena memory management, and distributed Context Parallelism.

---

🌟 Key Highlights

• Zero-Framework C++/CUDA Engine: Core tensor operations, backward autograd graphs, layer activations, and optimizers written in C++20 and CUDA 12+.
• 128k Context Window Support: Tiled FlashAttention with online Softmax in SRAM avoiding the $1.1\text{ TB}$ $O(N^2)$ memory bottleneck.
• Rotary Position Embeddings (RoPE): Configured with $\theta_{\text{base}} = 500,000.0$ for robust 131,072 token context extrapolation without phase collisions.
• Grouped-Query Attention (GQA): 32 Query Heads to 8 Key/Value Heads ($4:1$ ratio), cutting KV-cache memory bandwidth demand by $75%$.
• Custom Memory Arena: High-performance pooling and caching allocator eliminating runtime cudaMalloc synchronization overhead.
• Fused High-Throughput Kernels: Fused RMSNorm (warp-shuffle reduction), Fused SwiGLU, Fused Cross-Entropy with online LogSumExp, and Fused AdamW.
• Distributed Context Parallelism: Ring Attention over NCCL for multi-GPU long-sequence sharding.
• Mathematical Parity Verification: Verified against PyTorch golden references within $1\text{e-}5$ numerical tolerance.

---

📐 System Architecture

graph TD
    A["Raw Text Stream"] --> B["BPE Tokenizer (C++ Byte-Level)"]
    B --> C["Binary Dataset (mmap uint16)"]
    C --> D["DataLoader (Async Pinned Memory Stream)"]
    D --> E["Embedding Layer (Vocab: 32,000 -> 4096 Dim)"]

    subgraph "Transformer Decoder Stack (x32 Layers)"
        E --> F1["Pre-RMSNorm (Fused Warp-Shuffle Kernel)"]
        F1 --> F2["Q, K, V Projections (cuBLAS GEMM)"]
        F2 --> F3["RoPE Positional Embedding (theta=500k, 128k Context)"]
        F3 --> F4["Tiled FlashAttention (SRAM Online Softmax)"]
        F4 --> F5["Output Projection (Wo) + Residual Connection"]
        F5 --> F6["Pre-RMSNorm 2"]
        F6 --> F7["SwiGLU MLP (Gate * Up -> Down Projection)"]
        F7 --> F8["Residual Connection -> Layer Output"]
    end

    F8 --> G["Final RMSNorm"]
    G --> H["LM Head Projection (4096 -> 32,000 Vocab)"]
    H --> I["Fused Cross-Entropy Loss (Online LogSumExp)"]
    I --> J["Autograd Engine (Exact Analytical Backpropagation)"]
    J --> K["Fused AdamW Optimizer (Weight Decay + Cosine Warmup)"]
    K --> L["Binary Checkpoint (.bin / safetensors)"]

Loading

---

📂 Codebase Directory & File Architecture

An overview of every component and subsystem across the repository:

1. Header Interfaces (include/singularity/)

File	Subsystem	Responsibility
types.h	Type System	Defines data types (FLOAT32, FLOAT16, INT32), device targets (CPU, CUDA), and CUDA error handling macros (CUDA_CHECK).
config.h	Configuration	Hyperparameter structs (ModelConfig, TrainingConfig) and exact 7B (128k context) and 125M architecture presets.
memory/tensor.h	Tensor Core	Multi-dimensional tensor class with strided views, autograd tracking, slicing, and host/device memory semantics.
memory/memory_pool.h	Memory Arena	256-byte aligned caching memory pool and arena allocator for zero-overhead GPU memory reuse.
data/dataloader.h	Data Ingestion	Memory-mapped binary token dataset reader with randomized batch sampling and next-token target offsets.
tokenizer/bpe.h	Tokenizer	C++ Byte-Pair Encoding (BPE) subword tokenizer with vocabulary and merge-rule deserialization.
inference/kv_cache.h	Inference Engine	Pre-allocated multi-layer Key-Value Cache manager with dynamic sequence prefix slicing.
inference/sampler.h	Sampling	Temperature, Top-K, and Top-P (nucleus) multinomial token sampler.
distributed/nccl_comm.h	Distributed	Multi-GPU NCCL communicator singleton with automatic inter-process rendezvous.
distributed/ring_attention.h	Context Parallelism	Ring Attention manager for sharding 128k sequence contexts across multi-GPU clusters.
optim/adamw.h	Optimizer	Fused AdamW optimizer with decoupled weight decay, gradient clipping, and cosine warmup scheduler.
nn/*.h	Neural Layers	Class interfaces for Module, Linear, Attention (GQA), MLP (SwiGLU), TransformerBlock, and SingularityModel.
kernels/*.cuh	CUDA Kernels	CUDA kernel header declarations for FlashAttention, RMSNorm, RoPE, SwiGLU, Cross-Entropy, AdamW, and GEMM.

---

2. Execution Engines & CUDA Kernels (src/)

File	Subsystem	Responsibility
main.cu	CLI Application	Main binary entry point supporting training (train), attention benchmarking (benchmark), and generation (generate).
memory/tensor.cu	Tensor Operations	Native GPU memory allocation, host/device transfers, reshape/view logic, and gradient buffer management.
memory/memory_pool.cu	Memory Management	High-speed GPU arena caching pool eliminating runtime cudaMalloc / cudaFree synchronization bottlenecks.
kernels/flash_attention.cu	Attention Engine	Tiled SRAM FlashAttention forward and backward kernels with online softmax and register accumulation.
kernels/rmsnorm.cu	Normalization	Fused RMSNorm forward and backward kernels using single-pass warp-shuffle (__shfl_down_sync) reductions.
kernels/rope.cu	Positional Embeddings	Fused Rotary Positional Embedding (RoPE) forward and inverse rotations with $\theta = 500,000$.
kernels/swiglu.cu	Activations	Fused SwiGLU activation forward pass and analytical gradient backpropagation kernel.
kernels/cross_entropy.cu	Loss Engine	Fused Cross-Entropy loss with online LogSumExp and analytical gradient calculation in a single pass.
kernels/adamw.cu	Optimization	Fused AdamW GPU kernel updating weights, first momentum, and second momentum simultaneously.
kernels/gemm.cu	Matrix Multiplication	Thread-local cuBLAS GEMM wrapper and custom 2D tiled matrix multiplication kernels.
kernels/elementwise.cu	Elementwise Math	Fused GPU addition, in-place accumulation, and gradient scaling kernels.
nn/linear.cpp	Linear Projection	Fully-connected layer forward ($Y = X W^T$) and backward ($dX = dY W, dW += dY^T X$).
nn/attention.cpp	Attention Block	Grouped-Query Attention layer autograd orchestration (Q/K/V/O projections, RoPE, and FlashAttention).
nn/mlp.cpp	Feed-Forward Block	SwiGLU MLP layer autograd orchestration (Gate, Up, and Down projections).
nn/transformer_block.cpp	Transformer Block	Transformer Decoder Block autograd wiring with Pre-RMSNorm and dual residual skip pathways.
nn/model.cu	7B Transformer	Complete 7.24B model autograd graph, embedding kernels, forward loss, backward pass, and checkpoint serialization.
optim/adamw.cpp	Optimizer Logic	Global gradient norm clipping, learning rate warmup/decay scheduling, and parameter step orchestration.
distributed/nccl_comm.cu	Multi-GPU NCCL	Inter-GPU ncclAllReduce gradient synchronization, file rendezvous bootstrap, and stream management.
distributed/ring_attention.cu	Context Parallelism	Ring Attention token communication and online softmax LogSumExp rescaling across GPU ranks.
inference/kv_cache.cu	Inference Cache	Autoregressive Key-Value Cache state updates and dynamic sequence slicing.
inference/sampler.cu	Token Sampler	High-speed GPU Top-K, Top-P, and Temperature multinomial distribution token sampling.
tokenizer/bpe.cpp	BPE Tokenizer	Greedy Byte-Pair Encoding subword encoder/decoder with JSON vocabulary loading.
data/dataloader.cpp	DataLoader	Binary dataset file ingestion, memory-mapped streaming, and random batch extraction.

---

3. Training Scripts & Distributed Launchers (scripts/ & Root)

File	Responsibility
train.py	High-level Python launcher and orchestrator for training, benchmarks, and dataset preparation.
scripts/prepare_dataset.py	Downloads raw datasets, trains true subword BPE merges, and exports tokenized binary .bin files.
scripts/launch_distributed.sh	Single-node multi-GPU bash launcher configuring NCCL variables and spawning worker processes.
scripts/launch_multinode.py	Cloud multi-node distributed training orchestrator across GPU compute instances.

---

4. Tests & Performance Benchmarks (tests/ & benchmarks/)

File	Responsibility
tests/deep_math_audit.py	Microscopic gradient checking (finite difference) and RoPE 128k unitary orthogonality proofs.
tests/verify_against_pytorch.py	Mathematical parity suite comparing FlashAttention, RMSNorm, RoPE, and Cross-Entropy against PyTorch golden references.
tests/test_kernels.cu	C++/CUDA unit tests verifying individual kernel numerical accuracy and error bounds.
tests/test_autograd.cpp	End-to-end 2-layer autograd loss descent convergence verification test.
benchmarks/bench_attention.cu	128k FlashAttention latency (ms), effective compute throughput (TFLOPS), and memory avoidance profiler.
benchmarks/bench_throughput.cpp	Full training step tokens-per-second throughput profiler.

---

📊 7B Architecture Specifications

Hyperparameter	Value	Description
Model Type	singularity-7b	Autoregressive Decoder Transformer
Parameters	7.24 Billion	Exactly calibrated parameter profile
Hidden Dimension ($d_{\text{model}}$)	4096	Token hidden embedding dimension
Decoder Layers ($N_{\text{layers}}$)	32	Stacked Transformer blocks
Attention Heads ($N_q$)	32	Query attention heads ($d_{\text{head}} = 128$)
KV Heads ($N_{kv}$)	8	Grouped-Query Attention ($4:1$ GQA ratio)
SwiGLU Intermediate Dim ($d_{\text{mlp}}$)	14,336	Feed-forward hidden dimension ($\approx \frac{8}{3} d_{\text{model}}$)
Vocabulary Size ($V$)	32,000	Byte-level BPE vocabulary
Context Window ($S_{\text{max}}$)	131,072 (128k)	Maximum sequence length
RoPE Base Theta ($\theta$)	500,000.0	Long-context frequency base
RMSNorm Epsilon	1e-5	Numerical stability threshold

---

⚡ Mathematical Formulations & CUDA Fusions

1. 128k Context Tiled FlashAttention

Standard attention computes $A = \text{softmax}\left(\frac{Q K^T}{\sqrt{d}}\right) V$, which requires saving the $131,072 \times 131,072$ matrix ($1.1\text{ TB}$ of RAM).

SingularityEngine implements SRAM tiling ($B_r = 64, B_c = 64$) with online softmax: $$m_{\text{new}} = \max(m_{\text{old}}, \tilde{m}), \quad l_{\text{new}} = e^{m_{\text{old}} - m_{\text{new}}} l_{\text{old}} + \sum e^{\tilde{s}j - m{\text{new}}}$$ $$O_i = e^{m_{\text{old}} - m_{\text{new}}} O_i + P_{ij} V_j$$ Memory complexity is reduced from $O(N^2)$ to $O(N)$.

2. Fused RMSNorm with Warp Shuffles

$$\text{RMS}(x) = \sqrt{\frac{1}{d} \sum_{i=1}^d x_i^2 + \epsilon}, \quad y_i = \frac{x_i}{\text{RMS}(x)} \cdot \gamma_i$$ Reductions are executed entirely within warp registers using __shfl_down_sync intrinsics without writing intermediate stats to HBM.

3. Fused SwiGLU Activation

$$\text{SwiGLU}(x) = \left( \text{SiLU}(x W_{\text{gate}}) \odot (x W_{\text{up}}) \right) W_{\text{down}}, \quad \text{SiLU}(z) = \frac{z}{1 + e^{-z}}$$

---

🛠️ Build Instructions

Prerequisites

• CMake 3.20+
• C++20 compliant compiler (MSVC 2022 / GCC 11+ / Clang 14+)
• NVIDIA CUDA Toolkit 12.0+ (with cuBLAS)
• (Optional) Python 3.9+ with PyTorch for reference verification tests

Compilation

# Configure build directory
cmake -B build -DCMAKE_BUILD_TYPE=Release

# Compile all targets
cmake --build build --config Release

---

🚀 Running the Engine

1. Python Training & Verification CLI (Recommended)

# Run training step on full 7B 128k configuration
python train.py --config configs/7b_128k.json --seq_len 131072 --steps 100 --batch_size 1

# Launch 8x GPU distributed training via NCCL
python train.py --config configs/7b_128k.json --distributed --num_gpus 8

# Run golden mathematical verification audit
python train.py --verify

2. Native Bare-Metal C++/CUDA Binary Execution

# Run training step on micro-model (fast verification)
./build/singularity train --config micro --steps 100 --batch_size 4 --seq_len 512 --lr 3e-4

# Run training step on full 7B 128k configuration
./build/singularity train --config 7b --steps 1000 --batch_size 1 --seq_len 131072 --lr 3e-4

2. 128k Long-Context FlashAttention Benchmark

./build/singularity benchmark --seq_len 131072

3. Text Generation & Sampling

./build/singularity generate --weights checkpoints/step_500.bin --prompt "The future of artificial intelligence is"

4. Golden Reference Numerical Verification

python tests/verify_against_pytorch.py

---

💡 Architectural Decisions & Technical Rationale

1. Framework-Free C++/CUDA Core

Standard deep learning frameworks introduce dynamic memory fragmentation, operator dispatch overheads, and non-deterministic memory reuse patterns. By engineering a custom caching memory arena and fusing RMSNorm (warp-shuffles), RoPE, and SwiGLU directly into CUDA kernels, SingularityEngine eliminates intermediate HBM memory roundtrips and maximizes compute throughput.

2. 128k Long-Context Scalability

Standard attention mechanisms incur an $O(N^2)$ memory explosion ($1.1\text{ TB}$ for 128k context). SingularityEngine addresses this via:

1. Tiled FlashAttention: Keeps intermediate $Q K^T$ tiles strictly in high-speed on-chip SRAM with online softmax.
2. Grouped-Query Attention (GQA): $4:1$ query-to-KV head ratio reducing KV-cache bandwidth demand by $75%$.
3. Context Parallelism: Ring Attention over NCCL to shard 128k sequence chunks across GPU nodes.

3. Numerical Verification & Gradient Stability

All analytical kernel derivatives, autograd backward graphs, and fused activations are validated against finite-difference calculus to $10^{-10}$ precision, ensuring numerical stability during distributed multi-GPU execution.

---

📄 License

MIT License. Created by The Singularity AI Team.