Splitting GeoJSON and GeoPackage Without Loading Everything in Memory Using GeoSplit

Over the weekend, I downloaded farmland data for mainland Japan, which was about 50 GB. However, the file was far too large for my computer. I found Sebastien Tremblay’s GeoJSplit; however, it hasn't been updated since 2019. I wanted to expand that idea with exact-size splitting, validation, safer transactional output, progress reporting, a documented Python API, and GeoPackage support.

Introducing GeoSplit

To fix these issues, I’ve created GeoSplit. It is a Python CLI and library that:

• Streams and splits large GeoJSON files
• Splits GeoJSON by feature count or output size
• Splits GeoPackage layers by feature count
• Validates GeoJSON without creating output
• Preview split operations with a dry run
• Converts between GeoJSON and GeoPackage

For a multi-gigabyte input that may require significantly more memory than the file itself, GeoSplit streams through the features array. It keeps parser state and the active output chunk in memory rather than loading the complete FeatureCollection. This means memory usage is mainly determined by the selected chunk size, not the total size of the input file.

Installing geosplit

GeoSplit requires Python 3.10 or newer. Install the library from PyPI:

python -m pip install geosplit

To include GeoPackage support, install the optional dependency:

python -m pip install "geosplit[gpkg]"

Confirm installation:

geosplit --version
geosplit --help

Splitting GeoJSON by feature count

geosplit split farmland.geojson --features 10000

If farmland.geojson contains 45,000 features, Geosplit creates five GeoJSON files. The first four files will each contain 10,000 features. The final file contains the remaining 5,000. Each file can be opened or processed independently.

farmland_001.geojson
farmland_002.geojson
farmland_003.geojson
farmland_004.geojson
farmland_005.geojson

When no destination is specified, GeoSplit creates farmland_split beside the input. You can provide a different directory:

geosplit split farmland.geojson output --features 10000

Splitting GeoJSON by file size

geosplit split farmlands.geojson --size 10MB

Supported units include : B, KB, KiB, MB, MiB, GB, GiB

If farmlands.geojson is 45MB, GeoSplit creates five GeoJSON files. The first four files will each be 10MB. The final file contains the remaining 5. Each file can be opened or processed independently.

Splitting GeoPackage layer

geosplit split farms.gpkg --features 10000

If the source contains multiple layers, select one with --layer:

geosplit split map.gpkg --layer farms --features 10000

There are two important differences from GeoJSON splitting:

1. GeoPackage splitting currently supports only feature-count mode.
2. GeoPackage operations may load the selected layer through GeoPandas

Memory streaming applies only to GeoJSON splits, and GeoPackage splits may use more memory.

Previewing through dry run

geosplit split farmland.geojson --features 10000 --dryrun

geosplit split farms.gpkg --features 10000 --dryrun

A --dryrun or --dry-run returns filenames, feature counts, conflicts, and warnings without creating the output files. GeoJSON dry runs can also report expected output sizes. GeoPackage sizes are not estimated because the final size depends on how the layer is written.

Example outputs:

Source: farmland.geojson
Output: farmland_split
Features: 25,000
Total size: 13,482,107 bytes
Files:
  farmland_001.geojson: 10,000 features, 5,394,220 bytes
  farmland_002.geojson: 10,000 features, 5,381,442 bytes
  farmland_003.geojson: 5,000 features, 2,706,445 bytes

Source: farms.gpkg
Output: farms_split
Features: 25,000
Total size: not estimated
Files:
  farms_001.gpkg: 10,000 features, size not estimated
  farms_002.gpkg: 10,000 features, size not estimated
  farms_003.gpkg: 5,000 features, size not estimated
Warning: GeoPackage output sizes are not estimated during dry-run.

Validating GeoJSON

geosplit validate farmlands.geojson

For a valid file, the result will look something like:

Valid GeoJSON FeatureCollection
Features: 25,000
Geometry types:
  MultiPolygon: 20
  Polygon: 24,980
Null geometries: 0
Maximum nesting: 7
Coordinate dimensions: 2D
Coordinate precision: preserved

For invalid data, it reports the affected feature and path:

Invalid GeoJSON
Error: Feature 142 at geometry.coordinates[0][3]: polygon ring must be closed.
Features checked: 141For a machine-readable output, use --json . The result will be returned as JSON.

GeoSplit checks the JSON and GeoJSON structure, supported geometry types, coordinate nesting, finite numeric coordinates, polygon ring length and closure, trailing data, and excessive nesting.

This is a structural validation. It does not check geographic topology such as polygon self-intersections, and it does not modify or repair the input.

The validate command currently supports GeoJSON input. GeoPackage layers are checked during GeoPackage reading and splitting, but they do not use this validation report.

Converting GeoJSON and GeoPackage

After installing geosplit[gpkg], GeoSplit can convert between the two formats.

Convert GeoJSON to GeoPackage:

geosplit convert roads.geojson roads.gpkg

Choose the output layer name:

geosplit convert roads.geojson map.gpkg --output-layer roads

Convert a GeoPackage layer to GeoJSON:

geosplit convert map.gpkg roads.geojson --layer roads

Output monitoring and safe handling

Existing output is not overwritten unless --force is used.

geosplit split farmlands.geojson --features 10000 --force

GeoSplit only replaces files that it previously managed or recognizes as its output.

New files are prepared in a staging directory before existing managed files are replaced. If the process is interrupted during a transaction, GeoSplit attempts to recover it during the next run.

It also checks estimated disk requirements before beginning a split.

GeoSplit also displays progress during file splitting or validation:

Reading features      45,000 / 180,000
Writing chunks            12 / 48
Validating output          48 / 48

Use --quiet to suppress progress monitoring.

Using GeoSplit from Python

GeoSplit also provides a Python API. Check the API documentation for details.

Stream validated GeoJSON batches without writing files:

from geosplit import iter_batches

for batch in iter_batches("buildings.geojson", features=1000):
    print(len(batch["features"]))

Plan and execute a split:

from geosplit import plan_split, split_geojson

plan = plan_split(
    "buildings.geojson",
    features_per_file=1000,
)

print(plan.files)
print(plan.feature_count)
print(plan.conflicts)

result = split_geojson(
    "buildings.geojson",
    "output",
    features_per_file=1000,
)

print(result.files)
print(result.feature_count)
print(result.total_bytes)

Validate a file:

from geosplit import validate_geojson

report = validate_geojson("buildings.geojson")

if report.valid:
    print(f"Valid file with {report.feature_count:,} features")
else:
    for error in report.errors:
        print(error)

GeoPackage conversion is available from geosplit.convert

from geosplit.convert import (
    convert_file,
    plan_geopackage_split,
    split_geopackage,
)

Final thoughts

Splitting geospatial files becomes quite complicated once they are too large to handle as a single file.

GeoSplit is designed to keep that process predictable:

• Stream large GeoJSON files
• Split by feature count or exact size
• Split GeoPackage layers by feature count
• Validate before writing
• Preview operations with a dry run
• Protect existing output
• Convert between GeoJSON and GeoPackage
• Use the same functionality from Python

The project is open source and available on GitHub. The package can be installed from PyPI.

Bug reports, contributions, benchmark results, and feedback from real geospatial workloads are welcome!