itch_aggregator

GPU-accelerated Nasdaq ITCH 5.0 market-data pipeline. Streams a gzipped full-day TotalView-ITCH feed, parses it on the CPU, ships batches to an NVIDIA GPU over pinned memory, and maintains per-symbol VWAP, order imbalance, and quote-window bid/ask spread with CUDA kernels, emitting periodic CSV snapshots.

[ITCH .gz day file]
        |
        v
[CPU] gzFile stream  ->  framing (2-byte BE length)  ->  type dispatch
        |                  - 'R' builds locate->ticker table
        |                  - 'S'/'O' (market open) cuts the batch, schedules VWAP reset
        |                  - order hash resolves U/X/D order references
        v
[CPU] BatchAssembler: raw messages -> fixed 48-byte slots in pinned memory
        |
   cudaMemcpyAsync H2D (2 streams, 4-slot ring, event-gated recycling)
        |
        v
[GPU] decode kernel    : byteswap + field extraction -> SoA (1 thread/msg)
[GPU] aggregate kernel : per-symbol atomics (VWAP num/den, buy/sell vol,
                         cancel shares, window best bid/ask)
[GPU] snapshot kernel  : per-symbol metrics every N batches
        |
   cudaMemcpyAsync D2H (pinned, double-buffered)
        |
        v
[CPU] CSV rows for active symbols

Build

Requires CUDA 12.x, GCC 13+, CMake >= 3.25, zlib. Inside this repo's Zone container all of that is provisioned by zone.toml (CUDA 12.6, sm_89).

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

Data

curl -o data/12302019.NASDAQ_ITCH50.gz \
  "https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/12302019.NASDAQ_ITCH50.gz"   # 3.5 GB
python3 tools/make_fixture.py data/12302019.NASDAQ_ITCH50.gz data/fixture.itch.gz 20000000

The day file is always stream-decompressed; nothing larger than the fixture is ever materialized on disk.

No feed access? Generate a synthetic but wire-valid stream (correct framing and field layout, internally consistent order references) and run the entire pipeline against it:

python3 tools/make_synthetic.py data/synthetic.itch.gz 2000000
./build/itch_pipeline --file data/synthetic.itch.gz --no-csv --stats-json stats.json
python3 validation/validate_counts.py data/synthetic.itch.gz --against stats.json

Run

./build/itch_pipeline --file data/fixture.itch.gz > snapshots.csv
./build/itch_pipeline --file data/12302019.NASDAQ_ITCH50.gz --stats-json stats.json --no-csv
./build/itch_pipeline --file data/fixture.itch.gz --cpu-only        # CPU reference path

CSV columns: ts_ns,symbol,vwap,volume,buy_vol,sell_vol,imbalance,bid,ask,spread,trades,adds,cancel_shares. VWAP/volume/imbalance are cumulative since market open (System Event O); bid/ask are the best quotes added during the snapshot window.

Sample output (real rows, 2019-12-30 day file, late-session snapshot):

ts_ns,symbol,vwap,volume,buy_vol,sell_vol,imbalance,bid,ask,spread,trades,adds,cancel_shares
34886483801062,AAPL,289.2401,340591,71224,0,1.000000,287.4200,287.4300,0.0100,899,79215,32316
34886483801062,MSFT,158.8722,282848,62344,0,1.000000,157.8300,157.8400,0.0100,743,50844,7266
34886483801062,TSLA,423.5780,224745,152261,0,1.000000,415.0000,414.9700,-0.0300,2743,41077,6708
34886483801062,SPY,322.3129,108802,108802,0,1.000000,321.8800,321.8700,-0.0100,445,96912,375941

The crossed (negative) spreads on TSLA/SPY are the window-scoped bid/ask limitation in action, not a bug: without a full book, a window's best add-side bid and ask can straddle (see limitations below). VWAPs match the session (AAPL ~$289, TSLA ~$423) within the cumulative-window caveat.

Validation

Three independent layers:

1. Unit tests (ctest): framing across buffer boundaries, field offsets vs hand-built big-endian fixtures, order-hash backward-shift deletion, batch cutting at market open.

2. Bit-exactness: the GPU accumulators must equal the CPU reference implementation (src/pipeline/cpu_reference.cpp) exactly, over randomized synthetic streams (part of ctest) and over the real fixture.

3. Cross-implementation: validation/validate_counts.py re-parses the file in pure Python and diffs per-type message counts against itch_pipeline --stats-json:

   ./build/itch_pipeline --file data/fixture.itch.gz --no-csv --stats-json stats.json
   python3 validation/validate_counts.py data/fixture.itch.gz --against stats.json

Benchmarks

./build/parser_bench --file data/fixture.itch.gz   # CPU parse throughput (inflate excluded)
./build/kernel_bench --file data/fixture.itch.gz   # per-kernel us + GB/s via cudaEvents

Results on an RTX 4080 (Ada, sm_89), measured over a 20 M-message fixture carved from the Nasdaq 2019-12-30 TotalView-ITCH day file:

Metric	Value
Parser framing-only	231 M msg/s (7.0 GB/s)
Parser + hash + assembly (single core)	13.6 M msg/s
Messages shipped to GPU	91.9% (in-scope A/F/X/D/U/P/Q)
H2D bandwidth (pinned, 8192-msg batch)	22.9 GB/s
Decode kernel	1879 M msg/s (4.4 us/batch)
Aggregate kernel	2832 M msg/s (2.9 us/batch)
Snapshot kernel (16384 symbols)	7.8 us
Per-batch hot path (H2D+decode+aggregate)	25.0 us/batch (328 M msg/s)
End-to-end (parse+GPU aggregate, 20 M msgs)	2.84 s (7.05 M msg/s)

The single-core parser is the end-to-end bottleneck, exactly as predicted: at 13.6 M msg/s assembling but 7.05 M msg/s end-to-end, the wall-clock is dominated by gzip inflate plus framing, not by the GPU (the hot path sustains 328 M msg/s, ~46x the parser). All three target metrics in the spec are met: >5 M msg/s parse,

500 M ops/s aggregation, <50 us hot-path batch latency. Reproduce with parser_bench / kernel_bench above; numbers will vary with host CPU and the symbol skew of the input.

Design decisions and honest limitations

• Fixed 48-byte slots, not packed. Max in-scope message is 44 B; a fixed stride removes per-message offset bookkeeping and makes each slot three aligned 16-byte loads. The ~25% pinned-bandwidth waste is ~100 KB/batch.
• Big-endian conversion happens on the GPU. The single-core CPU parser is the pipeline bottleneck; it only inspects the type byte, the locate code and order references. Field byteswaps ride on thousands of GPU threads.
• Bid/ask is window-scoped, not an order book. Best bid/ask are the extremes of add/replace flow within a snapshot window (atomicMax/atomicMin), reset at each snapshot. Cancels cannot lower the best bid without per-price book state, so a window's "spread" can be stale or crossed. A real spread needs a full book; this is documented scope, not an oversight.
• Imbalance uses P-message sides only. The B/S indicator on a non-cross trade describes the resting non-display order, so this is resting-side imbalance, not aggressor imbalance (which would need E/C executions, out of scope). Cross trades (Q) count toward VWAP/volume but are imbalance-neutral.
• Order hash on the CPU. Open addressing, linear probing, backward-shift deletion, 2^24 slots (~400 MB). Replaces whose original order predates the stream are shipped with an unknown-side tag and counted (unresolved_replaces in stats).
• Atomic contention. Aggregation uses one global atomic per message-field; a planned optimization (after profiling) is warp-level pre-aggregation with __reduce_add_sync for hot symbols.

Layout

src/parser/    message_types.hpp (wire layout), itch_parser (framing), order_hash
src/pipeline/  batch_assembler, ring_buffer (pinned slot ring), cpu_reference, snapshot.hpp
src/cuda/      kernels.cuh, decode_kernel.cu, aggregate_kernel.cu, snapshot_kernel.cu
src/main.cpp   CLI + double-buffered stream loop
tests/         doctest suites + StreamBuilder fixture builder
benchmarks/    parser_bench (CPU), kernel_bench (cudaEvent timings)
validation/    validate_counts.py (independent Python cross-check)
tools/         make_fixture.py (carve a fixture from a day file), make_synthetic.py (generate one)