Uncaught Exception: RangeError: Invalid string length

[CiaranOMara]

I'm getting error "Uncaught Exception: RangeError: Invalid string length".

This error comes about when loading the following data with julia> data |> Voyager().

Is there an upper limit on what Voyager can handle? Any assistance would be much appreciated.

[davidanthoff]

I can replicate that on my system with a lot of RAM.

In general the transmission of data sets is quite inefficient right now: on the julia side, we write the table out as a JSON string, then send the JSON string via a named pipe to the Electron main process, and then pass it on to the Electron window process, and there it then gets handled by the voyager JavaScript code. My best guess is that somewhere along that path, there is some piece of code that we rely on that doesn't handle strings well that are on the order of a couple of GB...

I'm not sure right now what the best strategy is... One would be to go through the code in https://github.com/davidanthoff/Electron.jl and try to figure out where exactly this problem is happening, and see whether we can find a workaround. On some level, even though our current architecture is not the most efficient, it should still work on a system that has enough RAM.

The other, more long term option would be to rethink how we shuffle around the data in the first place. The vega team is looking into things like Arrow support, and I think some of that might be really promising...

I guess another option would be to add the ability to sample from the source data and only send fewer rows to voyager, but that also doesn't seem great...