Memory Usage

[nickos556]

Great tool, thanks so much for sharing!
I am running helix-trace using the following command line:

./helix-trace -i /media/sf_data/ITCH5.0/08302017.NASDAQ_ITCH50 -s AAPL -P nasdaq-binaryfile-itch50 -f csv -o AAPL.csv

08302017.NASDAQ_ITCH50 is downloaded from ftp://emi.nasdaq.com/ITCH/ and is roughly 5.4gb

When I run the tool, it steadily uses more and more memory, until it crashes (I only have 4gb of memory). I am running using Fedora 26. Any ideas what is going on? Does helix not delete orders after they are executed or cancelled?

thanks
Nick

[penberg]

@nickos556 I'm able to process a similarly sized file, but I also have 32 GB of RAM. Could you please try reproducing with a debug build that enables AddressSanitizer?

mkdir -p build/debug
cd build/debug
cmake -DCMAKE_BUILD_TYPE=debug ../..

[penberg]

@nickos556 I ran helix-trace using Massif and the peak memory usage is around 2.8 MiB for a 7.1 GiB ITCH 5.0 file. The memory usage increase you're seeing is an artifact of helix-trace using mmap() for accessing the file but the kernel will manage that memory for you and shouldn't cause you to run out of memory.