Improve parsing speed

[jasperhabicht]

Check the parsing algorithm for improvements. Especially:

• parsing and storing of whole objects and arrays,
• checking for valid strings (currently done via inserting scan marks).

[jasperhabicht]

The current behaviour of storing whole objects and arrays (and even the complete original JSON string) as values in a property list may well lead to memory issues for large data.

A suggestion would be to add a key to disable storing of full objects and arrays.

[jasperhabicht]

With version 1.4.0, the key skip structures can be set in order for objects and arrays not to be stored as values to the relative values (including the whole JSON string as value to the key .). As it is often not necessary to store whole objects or arrays, parsing speed can be increased this way.

In addition collection of arrays and objects has been optimised, so parsing speed should have increased considerably. Instead of counting all tokens, only braces and brackets are now counted via a counter (except those within strings), so a balanced group can be selected easily given a valid input.

Benchmark testing has been implemented (solving #62) proving the huge improvements regarding parsing speed between version 1.3.1 and 1.4.0.

[jasperhabicht]

\__jsonparse_collect_object_array:Nn can probably be optimised so that it does not parse one token at a time but grab everything before the next [ or { in one go.

[jasperhabicht]

Some instances of e type expansion can be avoided which will improve parsing speed.

Where possible, assignments should be reduced.

[HowcanoeWang]

Parsing 6k characters json cost over 3 minutes, and results in over 10 mins compile time for xelatex -> biber -> xelatex*2 complier process on my Intel 13900K

[jasperhabicht]

Parsing 6k characters json cost over 3 minutes, and results in over 10 mins compile time for xelatex -> biber -> xelatex*2 complier process on my Intel 13900K

Thank you for your comment. I know that parsing large or complex data can take a while which is why I track parsing speed with a medium-sized example file (you can see the results in the benchmark folder in the main tree; the file has about 4k6 chars and parsing currently takes about 1 second).

Here are some ideas to improve the parsing speed:

1. Make sure that you update jsonparse to the most recent version. Parsing time was reduced a lot with version 1.4.0 and again a bit with version 1.5.0.
2. You can try to add skip structures as option to the \JSONParse(FromFile) command. So, for example, \JSONParse[skip structures]{\myjson}{ { "foo" : "bar" } } should be considerably faster than \JSONParse{\myjson}{ { "foo" : "bar" } }. But this can, of course, only make sense if you don't need to access the full objects or arrays later on in your code.
3. You can try to add validate numbers=false as option to the \JSONParse(FromFile) command. So, for example, \JSONParse[validate numbers=false]{\myjson}{ { "foo" : 1.23 } } should be a tad faster than \JSONParse{\myjson}{ { "foo" : 1.23 } }, but the effect will only be visible if the data acutally contains a larger amount of numbers.
4. If the JSON data does not change often, you can externalise parsing which will reduce compilation time as the data is only parsed once. This is done by adding the externalize key to the \JSONParse(FromFile) command, such as: \JSONParse[externalize]{\myjson}{ { "foo" : "bar" } }.
5. If nothing helps, you may want to use Lua as parser. I did not yet implement it to the package, but you can make use of the JSON parser implemented in Lua provided by ConTeXt.

For your concrete issue, you should in fact first try to update. A parsing time of >3 minutes for 6k chars seems in fact unusual for a current version of the package. If you still face issues, you are welcome to ask for help on https://tex.stackexchange.com or send me your JSON file along with the log file.

[HowcanoeWang]

@jasperhabicht thanks for this kindly advice, I updated to the latest version released 3 days ago (1.6.2), the speed increased dramatically and became acceptable!

[jasperhabicht]

Can we further improve parsing speed by reducing the use of \tl_put_right:Nn?

This is already quite fast, but the issue is that it is called too often, especially for collecting arrays and objects.

Is there a better way to grab arrays or objects at once? Maybe by injecting a token at the start or copying the current string start starts with the relevant array or object and clip away the rest?

Can we reconstruct any array or object from the already parsed parts? If this can be done expandable, there would be no need to collect full arrays or objects in the first place.

[jasperhabicht]

Instead of \tl_put_right:Nn, we can build a token list using the \tl_build_... functions. These should be a lot faster than the regular \tl_put_... functions.

Changed for v1.7.3: small improvement of parsing speed.