Can we stream large JSON files?

[nkinesis]

Problem: We have to deal with JSON files which are several GBs large, and we cannot afford to load the entire JSON into RAM every time we want to perform operations with it.

Action: See if you can find any libraries that do JSON streaming. You can also explore other streaming formats similar to JSON, such as XML.

[nkinesis]

@SikandarEjaz

[nkinesis]

Can you please mention the libraries you are experimenting with @SikandarEjaz ?

[SikandarEjaz]

There are libraries designed to handle multi-gigabyte JSON files efficiently. I'm currently experimenting with the Jackson Streaming API, newline-delimited JSON (NDJSON), and simdjson.