Goblin is a CLI tool for extracting data from ArcGIS Map/Feature Servers. It is written in Go with concurrency in mind, so it excels in extracting relatively large datasets (>1GB). Goblin was created as an alternative to GDAL's ESRIJSON driver. We found that ogr2ogr struggles particularly with larger datasets due to its non-concurrent nature and the ArcGIS default record limit (usually 1-2K), which results in very long download times and/or disconnection from the server.
The easiest way to install on MacOS is via Homebrew. Firstly, tap the third-party repository with brew tap kryn3n/kryn3n. You can read more about third-party repositories here.
Now we can install the program as usual with brew install kryn3n/kryn3n/goblin.
The goblin command is run with 2 required positional arguments, plus 3 optional flags. The first positional argument should be the URL to the MapServer or FeatureServer, the second should be the output filename (recommended to use .geojsonl as Goblin creates a newline-delimited GeoJSON file. You can read more about the format here). The optional flags can be found below.
| Flag | Description | Required | Default |
|---|---|---|---|
-c |
Concurrency limit | No | 100 |
-l |
Layer ID | Yes | -1 |
-m |
Max. record limit | No | 1000 |
If no layer ID is supplied, you will be prompted for it after running your command.
For this benchmark we are using the cadastre layer from Queensland's Land Parcel Property Framework server. The layer contains about 3.4 million records and the resulting file once downloaded is over 3.5GB.
Goblin downloaded the same data in less than 2 mins vs. GDAL's 105 mins. That's over 60x faster! 🚀 Results may vary depending on internet connection and the environment where the commands are being run so we encourage you to try the below commands for yourself.
time goblin -l 4 \
"https://spatial-gis.information.qld.gov.au/arcgis/rest/services/PlanningCadastre/LandParcelPropertyFramework/MapServer" \
test.geojsonl
________________________________________________________
Executed in 103.10 secs fish external
usr time 252.44 secs 0.25 millis 252.44 secs
sys time 23.99 secs 1.07 millis 23.99 secstime ogr2ogr test.geojsonl \
-f "GeoJSONSeq" \
"https://spatial-gis.information.qld.gov.au/arcgis/rest/services/PlanningCadastre/LandParcelPropertyFramework/MapServer/4/query?where=1=1&returnGeometry=true&outFields=*&orderByFields=objectid&f=geojson"
________________________________________________________
Executed in 105.17 mins fish external
usr time 512.13 secs 715.00 micros 512.13 secs
sys time 10.11 secs 826.00 micros 10.11 secs