Conversion of big Turtle file failes with `RangeError: Invalid string length`

[hexagon6]

Using a very big ttl file (in my case 700MB) causes cli to fail when trying to convert lines.join("\n")

Steps to reproduce:

$ npx
$ ttl2jsonld esco.ttl > esco.jsonld
/home/redacted/project/node_modules/.pnpm/@frogcat+ttl2jsonld@0.0.10/node_modules/@frogcat/ttl2jsonld/bin/cli.js:16
  const turtle = lines.join("\n");
                       ^

RangeError: Invalid string length
    at Array.join (<anonymous>)
    at Interface.<anonymous> (/home/redacted/project/node_modules/.pnpm/@frogcat+ttl2jsonld@0.0.10/node_modules/@frogcat/ttl2jsonld/bin/cli.js:16:24)
    at Interface.emit (node:events:519:35)
    at Interface.close (node:internal/readline/interface:559:10)
    at ReadStream.onend (node:internal/readline/interface:273:10)
    at ReadStream.emit (node:events:507:28)
    at endReadableNT (node:internal/streams/readable:1701:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:90:21)

Node.js v24.2.0

[frogcat]

Hi, sorry for the very late response — I completely missed this issue. Thanks for your patience.

Regarding the problem you’re seeing:

• Very large strings (over ~700MB) are often not safely handled in JavaScript environments.
  In practice, JavaScript strings tend to hit limits around 512MB, depending on the engine.
  For reference, see the MDN documentation on String.length:
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length

• A common approach to dealing with this limitation in JavaScript is to process data as streams instead of loading it as a single string.
  However, ttl2jsonld is not planned to support streaming input/output, so this approach is not available in this tool.

As a workaround, please consider splitting the Turtle file into smaller chunks before converting, or processing it in multiple parts.

Sorry again for the delayed response, and thanks for raising the issue.

[hexagon6]

Thank you @frogcat , the workaround is a good idea.
Do you think the resulting json parts can be merged easily? I assume I'd have to split it with complete subject parts for it to still be considered fully.

[frogcat]

Hi @hexagon6 , here is a possible workaround using shell commands.

For example, given the following Turtle files:

# a.ttl
@prefix foaf: <http://xmlns.com/foaf/0.1/> .

<#Alice> a foaf:Person ; foaf:name "Alice" .
<#Bob> a foaf:Person ; foaf:name "Bob" .
<#Charlie> a foaf:Person ; foaf:name "Charlie" .

# b.ttl
@prefix foaf: <http://xmlns.com/foaf/0.1/> .

<#David> a foaf:Person ; foaf:name "David" .
<#Eve> a foaf:Person ; foaf:name "Eve" .

You can generate a merged JSON-LD document like this:

$ ttl2jsonld a.ttl | sed '$d' | sed '$d' | sed '$s/$/,/' > merged.json
$ ttl2jsonld b.ttl | sed '1,/@graph/d' >> merged.json

This produces a valid merged JSON-LD document such as:

{
  "@context": {
    "foaf": "http://xmlns.com/foaf/0.1/"
  },
  "@graph": [
    {
      "@id": "#Alice",
      "@type": "foaf:Person",
      "foaf:name": "Alice"
    },
    {
      "@id": "#Bob",
      "@type": "foaf:Person",
      "foaf:name": "Bob"
    },
    {
      "@id": "#Charlie",
      "@type": "foaf:Person",
      "foaf:name": "Charlie"
    },
    {
      "@id": "#David",
      "@type": "foaf:Person",
      "foaf:name": "David"
    },
    {
      "@id": "#Eve",
      "@type": "foaf:Person",
      "foaf:name": "Eve"
    }
  ]
}

This approach assumes that a.ttl and b.ttl share the same @prefix definitions, and that each file contains multiple statements (i.e., @graph is generated in the output).

I hope this helps!