"Invalid string length" when merge a large files

[anphanngocvn]

I have a few *.json files (some of them about 750Mb) need to be merged into one *.geofile. However when I run this command to merge them. geojson-merge folder/*.geojson > combined.geojson. I have following errors:

`/geojson-merge/geojson-merge:17
process.stdout.write(JSON.stringify(merge(argv._.map(function(n) {
^

RangeError: Invalid string length
at JSON.stringify ()
at Object. (/geojson-merge/geojson-merge:17:29)
at Module._compile (internal/modules/cjs/loader.js:701:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
at Module.load (internal/modules/cjs/loader.js:600:32)
at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
at Function.Module._load (internal/modules/cjs/loader.js:531:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)`

How do you tackle this?

[gorbypark]

Hi, I ran into the same problem a few weeks ago. The issue is that you can only add so many arguments to a command (maybe 1000 on bash/linux? I can't remember,). The only solution to this is to do a batch operation (for example, if you have 10,000 files to merge, merge 500 at a time until you have 20 files, and then merge those 20 together). The solution I took was to write a quick node script that iterated over every file in a folder and merges that way. There is not really a max file type restriction in node when iterating over a folder (although you could risk running out of ram!).

[qrcf]

I have the same issue on the entire Kantō region (roads only) of Japan. Its just merging two files. not hundreds. Apparently the file itself is too big at 769.3mb

[warrenca]

@anphanngocvn @qrcf which version of node are you using? i encountered the problem in v6.10, tried in v11 and it's all working fine. I still trying to find out now which node version was it really fixed.

Might want to remove the 2 from

geojson-merge/geojson-merge

Line 19 in fccdc8d

})), null, 2));

, indentation seems to inflate file sizes.

[jackcjp]

Might want to remove the 2 from

geojson-merge/geojson-merge

Line 19 in fccdc8d

})), null, 2));

, indentation seems to inflate file sizes.

Change 2 to 0 works for me! thank you @cagriozkurt