Repository navigation
Structure Overview
- A Static Memory Arena to keep latency flat.
- A Non-blocking Stream Scheduler for low overhead.
- A Shared-memory Tiled INT8 GEMM Kernel for fast hardware math.
- A Main test runner and a CMake build pipeline to tie it all together. extra
- Fused ReLU Activation Function (as one of the most powerful tricks in systems engineering)
- Custom memory allocator feature
- Benchmarking to CSV
- Tests
Here is the project structure overview and functional description, followed by the complete, implementation of Deterministic Static Memory Allocator (CudaScratchpadAllocator).
This allocator pre-allocates a single contiguous block of GPU memory during initialization. During inference, it chunks out memory offsets with zero runtime cudaMalloc calls, achieving completely deterministic execution latency.
files: include/allocator.hpp and src/allocator.cu
This design uses a Monotonic Arena (Scratchpad) Allocator pattern. It guarantees O(1) allocation and reset times, perfect for inference loops.
- Explicit Alignment Control: CUDA requires proper alignment (often 256 bytes for performance optimizations like coalesced memory access). This code implements custom alignment offsets manually on raw pointers.
- RAII Design Principles: The memory space lifecycle safely adheres to strict C++ memory design practices (rule of 5/rule of zero handling explicitly via move operators).
Here is the implementation of the Hardware-Bound Task Scheduler (CudaStreamScheduler).
This scheduler encapsulates CUDA Streams for non-blocking asynchronous execution queues and uses CUDA Events for microsecond-accurate hardware-level profiling and dependency tracking—completely bypassing host-side thread management overhead.
files: Header Interface: include/scheduler.hpp and Implementation: src/scheduler.cu
- cudaStreamNonBlocking Flags: By default, standard CUDA streams synchronize with the legacy default "NULL" stream. Using non-blocking flags separates your execution lanes completely, allowing true, asynchronous runtime concurrency.
- Zero-Overhead Host Profiling: Standard CPU clocks (std::chrono) introduce PCIe latency and CPU execution bubbles when trying to profile GPU work because they force full device synchronization. Using native cudaEventElapsedTime tracks execution timestamps natively inside the hardware command queue itself.
Here is the implementation of a custom, high-performance INT8 Quantized GEMM (General Matrix Multiplication) CUDA Kernel. The kernel-level systems, this implementation goes beyond a naive nested loop. It implements Shared Memory Tiling (Blocking) and uses Vectorized Memory Access (int4 / char4) to maximize memory bandwidth and keep the streaming multiprocessors (SMs) completely saturated.
files: Header Interface: include/kernels.cuh and Implementation & Architecture: src/kernels.cu
We will use a tile dimension of 32. This means each block handles a 32 × 32 output region, caching pieces of data inside high-speed GPU Shared Memory to avoid hitting global memory repeatedly.
- Memory Coalescing & Tiling: Instead of fetching single elements from high-latency global memory O(K) times, threads load contiguous blocks into high-speed local static shared caches. This yields massive memory performance gains.
- #pragma unroll Vector Optimization: Telling the NVCC compiler to unroll the inner mathematical reduction tells the compiler to flatten branches, keeping the GPU instruction pipeline completely full.
- Fused De-quantization: The kernel handles integer math internally and dynamically converts it back into FP32 floats alongside bias adding (C[i] = accum * alpha + bias) right before saving. This cuts out an entire extra kernel pass from memory.
Here we're creating the implementation of the Unified Runtime Engine Framework (Engine) to orchestrate these pieces together This class orchestrates all the underlying systems components we have built: it accepts weight handles, configures the execution layers, manages the lifecycle of input/output bindings, resets the CudaScratchpadAllocator instantly between runs, and pushes operations asynchronously through the CudaStreamScheduler.
files: Header Interface: include/engine.hpp and Implementation: src/engine.cpp
- True Non-Blocking Pipelines: Notice that Engine::forward dispatches instructions directly to the cudaStream_t pipeline. The CPU thread returns immediately after queuing the work, bypassing synchronization bottlenecks.
- Predictable, Fixed Layouts: Memory footprints are zeroed out instantly using allocator_.reset() at the start of each execution boundary. Regardless of how many frames pass through the engine, the footprint remains entirely flat.
This file serves as the main entry point and benchmark test suite for your engine. It generates synthetic quantized weights and input tensors, registers a multi-layer network architecture, executes a warmup run, and runs a high-precision performance benchmark. This setup will gives you to tools to measure the latency gains.
file: Implementation: src/main.cpp
- Warm-up Strategy: Hardware drivers scale GPU cores up and down to save power. Running a hidden warm-up pass before starting the loop makes sure the graphics card is locked at peak performance state, removing artificial anomalies from the benchmark data.
- Microsecond Precision Tracking: By running the loop 1,000 times using the hardware engine events we implemented earlier, the code yields highly accurate runtime benchmarks.
MakeLists.txt file to handle NVCC compilation flags and clean up the workspace configuration.
It handles cross-platform detection of the NVIDIA CUDA Toolkit, configures the proper C++ standard, and instructs the compiler to generate heavily optimized machine code tailored for NVIDIA GPUs.
File: Implementation: CMakeLists.txt
How to Compile and Run Your Project, please read here: How to run
Here is how you can implement a Fused ReLU Activation Function directly into custom INT8 GEMM kernel. Instead of writing matrix outputs back to high-latency Global Memory (VRAM) only to immediately reload them into registers for a separate ReLU step, we compute and apply the activation in-register while the tensor data is already loaded in the Streaming Multiprocessor (SM). This effectively makes the activation phase free from a latency perspective.
file: Updated Header Interface: include/kernels.cuh
It required to update the kernel launcher to include a boolean flag fuse_relu so your runtime engine can dynamically toggle the activation step depending on the layer's position in the network.
file: Updated Kernel Implementation: src/kernels.cu
We modify the end of the tiled matrix multiplication kernel to conditionally clamp negative values to zero using the high-performance hardware-mapped fmaxf() function right before the values are written out to global memory.
file: Update to the Framework Layer Layout: include/engine.hpp
We will now add a fuse_relu flag to our layer parameters so our setup functions can capture this structural constraint.
file: Update inside src/engine.cpp
Adjusting the Model Design with Fused ReLU in src/main.cpp
Replace actual ngine.add_linear and uncommentiong code below we can explicitly configure our hidden layers to use Fused ReLU while keeping the final output layer linear.
// Update the engine registration setup lines in src/main.cpp:
// Layer 1 targets hidden outputs -> We fuse ReLU activation here!
engine.add_linear_layer(BATCH_SIZE, HIDDEN_DIM, INPUT_DIM, 0.001f, true, d_w1, d_b1);
// Layer 2 is our network head/classifier classification score output -> Keep ReLU false
engine.add_linear_layer(BATCH_SIZE, OUTPUT_DIM, HIDDEN_DIM, 0.002f, false, d_w2, d_b2);Standard framework pipelines (like basic unoptimized PyTorch scripts) launch a GEMM Kernel, wait, and then launch a separate vector Elementwise ReLU Kernel. This solution memory-bandwidth bound operations and shows how to reduce expensive memory-roundtrips by fusing logic directly onto active processing hardware.
Here is the implementation of an advanced custom memory allocator feature: Ping-Pong Arena Allocation (Double-Buffering).
In a multi-layer neural network execution loop, the output of Layer N becomes the input to Layer N+1. A naive runtime engine creates a new allocation for every single layer's output tensor, resulting in memory fragmentation and constant pointers tracking.
A Ping-Pong Allocator solves this elegantly at zero overhead. It creates exactly two static memory buffers (Buffer A and Buffer B). Layer 1 reads from the raw input and writes to Buffer A. Layer 2 reads from Buffer A and writes to Buffer B. Layer 3 reads from Buffer B and writes back to Buffer A. This completely eliminates dynamic indexing, guarantees zero memory leaks, and caps the maximum engine scratchpad size to exactly the largest two adjacent layers in the network.
file: Header Interface Update: include/allocator.hpp
We will extendyour original CudaScratchpadAllocator to support ping-pong buffer management without changing its underlying static monolithic allocation philosophy.
file: Implementation: src/allocator.cu
file: Updating the Pipeline Orchestration: src/engine.cpp
Here it becomes as "Engine::forward". The complex pointer slicing logic from before is completely replaced by this deterministic ping-pong toggle sequence.
- Cache Locality Awareness: Reusing the exact same two blocks of device pointers ensures the GPU L2 caches can aggressively intercept and accelerate global memory transfers between layers.
- Predictable Architecture: The system uses a strict bounding layout limit, eliminating the need to track varying internal layer allocation offsets entirely.
Benchmarking scripts to export your results to a CSV file.
To keep your code completely framework-free and zero-overhead, this implementation uses native C++ file streams (std::ofstream) to write the data. The logging happens outside the hot path (after the loops finish), ensuring that writing to the hard drive does not artificially slow down or ruin your ultra-low latency GPU benchmarks.
file: Updated Interface: include/engine.hpp
We will add a helper struct and a tracking method to the Engine class so it can return its exact execution dimensions. This lets the benchmarking script record the model configuration alongside the latency results.
file: Updated Benchmarking Suite: src/main.cpp
We update src/main.cpp to run the 1,000-iteration test loop, store the raw latency of every single run into a pre-allocated host vector, and then gracefully flush the data into a structured CSV file.
What Output Look Like (inference_benchmarks.csv)
iteration,latency_ms,latency_us,total_layers,scratchpad_capacity_bytes
0,0.0124,12.4,2,4096
1,0.0121,12.1,2,4096
2,0.0122,12.2,2,4096
...
999,0.0121,12.1,2,4096
- No IO Bloat in Hot Path: Notice that the script never runs csv_file << ... inside the loops. Doing I/O operations inside timed loops introduces unpredictable operating system context switches. Pre-allocating data to the host memory array (run_latencies_ms) guarantees your measurement loop remains strictly clean.
- Ready for Visualization: Visitors checking out our GitHub can download this CSV directly and drop it into a Jupyter Notebook, Pandas, or Excel to draw scatter plots showing the p99 latency consistency of your custom inference engine.
Adding an automated test suite, for systems projects built on raw hardware control, tests verify mathematical correctness (making sure optimizations didn't break the actual output) and catch edge cases (like buffer sizes or invalid shape inputs) before runtime execution crashes.
To keep our project clean, modular, and easy to reproduce, we can use Catch2 (v2) or Google Test. Catch2 is excellent for our GitHub because it can be integrated as a single header file without making users download heavy system dependencies.
files: dedicated tests/,
file: Implementation: tests/test_engine.cpp
This test suite covers two critical validation vectors: verifying that the custom allocator handles errors predictably, and verifying that the fused ReLU kernel logic functions perfectly.
file: Integrated Build Configuration: CMakeLists.txt
This is build configuration to declare a standalone run_tests binary alongside the main benchmark engine target.
- Mathematical Validation: Optimizing CUDA kernels with shared memory blocking frequently introduces off-by-one errors or indexing bugs. It can be extended with other mathematical validation test suites proves our custom kernels compute accurately.
- Defensive Engineering: Testing out-of-memory errors on the scratchpad shows us how to safely manage bare-metal hardware resources without creating silent pointers or data corruption bugs.