[v0.21.0] pyogrio generator

[perrygeo]

fiona is a great library that worked well with our records-based iteration approach. However, recent releases don't have a Python 3.14 binary wheel which complicates installation for many users. Additionally several projects, notably geopandas, are switching to pyogrio for bulk loading performance.

This PR implements pyogrio as the default rasterstats engine.

• A pip install rasterstats will include pyogrio by default
• A pip install rasterstats[fiona] will let you specify the engine=fiona parameter if you need to keep the old behavior.
• Tests that require fiona will be skipped if it isn't importable. Fiona is completely optional.
• Performance differences:
  • Most use cases and formats will see an improvement in performance, accompanied by an increase in memory usage for batching IO into chunks.
  • Very large GeoJSON files are an exception, and are likely to get slower the larger they are. Use engine=fiona if you absolutely must get top performance out of very large 50MB+ GeoJSONs.

[perrygeo]

With chunk_size=24_000

$ uv run scripts/bench_engines.py 500000
Generating 500,000 random point features over slope.tif …
Wrote 65.7 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpxsrljdmu.geojson

fiona  : 4.539s  (110,158 feat/s)
pyogrio: 22.134s  (22,590 feat/s)

fiona is 4.88x faster

[perrygeo]

with recent changes:

$ uv run scripts/bench_engines.py 500000
Generating 500,000 random point features over slope.tif …
Wrote 65.7 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpgmt3f3bx.geojson

fiona  : 4.514s  (110,767 feat/s)
/Users/matthew/projects/python-rasterstats/.venv/lib/python3.14/site-packages/pyogrio/raw.py:198:
 RuntimeWarning: driver GeoJSON does not support open option USE_ARROW
  return ogr_read(
pyogrio: 10.205s  (48,997 feat/s)

fiona is 2.26x faster

[perrygeo]

Now we're getting somewhere...

uv run scripts/bench_engines.py 500000
Generating 500,000 random point features over slope.tif …
Wrote GeoJSON    65.7 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmp4dg3z8wh.geojson
Converting to GeoPackage via ogr2ogr …
Wrote GeoPackage 44.9 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpwn09reve.gpkg

=== GeoJSON ===
fiona  : 4.635s  (107,868 feat/s)
pyogrio: 10.397s  (48,089 feat/s)

fiona is 2.24x faster (GeoJSON)

=== GeoPackage ===
fiona  : 2.858s  (174,936 feat/s)
pyogrio: 2.050s  (243,951 feat/s)

pyogrio is 1.39x faster (GeoPackage)

Some formats get faster, some slower...

[perrygeo]

uv run scripts/bench_engines.py 500000
Generating 500,000 random point features over slope.tif …
Wrote GeoJSON    65.7 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmp8re3ju89.geojson
Converting to GeoPackage via ogr2ogr …
Wrote GeoPackage 44.9 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmp1vcc72nk.gpkg

=== GeoJSON ===
fiona  : 4.556s  (109,738 feat/s)
pyogrio: 8.553s  (58,459 feat/s)

fiona is 1.88x faster (GeoJSON)

=== GeoPackage ===
fiona  : 2.805s  (178,256 feat/s)
pyogrio: 0.397s  (1,260,567 feat/s)

pyogrio is 7.07x faster (GeoPackage)

[perrygeo]

uv run scripts/bench_engines.py 500000
Generating 500,000 random point features over slope.tif …
Wrote GeoJSON    65.7 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpwhtavcrw.geojson
Converting to GeoPackage via ogr2ogr …
Wrote GeoPackage 44.9 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpwurcens9.gpkg
Converting to Shapefile via ogr2ogr …
Wrote Shapefile  21.9 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpuujs_7lq
Converting to Parquet via ogr2ogr …
Wrote Parquet    19.8 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpe5yn18ka.parquet

=== GeoJSON ===
fiona  : 4.541s  (110,112 feat/s)
pyogrio: 8.623s  (57,984 feat/s)

fiona is 1.90x faster (GeoJSON)

=== GeoPackage ===
fiona  : 2.793s  (179,037 feat/s)
pyogrio: 0.399s  (1,251,842 feat/s)

pyogrio is 6.99x faster (GeoPackage)

=== Shapefile ===
fiona  : 3.231s  (154,756 feat/s)
pyogrio: 0.533s  (938,250 feat/s)

pyogrio is 6.06x faster (Shapefile)

=== Parquet ===
fiona  : 2.820s  (177,288 feat/s)
pyogrio: 0.365s  (1,369,180 feat/s)

pyogrio is 7.72x faster (Parquet)

[perrygeo]

The latest benchmark, fiona and pyogrio are working. The pyogrio generator is now performing well enough for me to use it as the default!

uv run scripts/bench_engines.py
Generating 100,000 random point features over slope.tif …
Wrote GeoJSON    13.1 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmp_brzukhr.geojson
Converting to GeoPackage via ogr2ogr …
Wrote GeoPackage 9.0 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmp7ptsired.gpkg
Converting to Shapefile via ogr2ogr …
Wrote Shapefile  4.4 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmpa_uol1xl
Converting to FlatGeobuf via ogr2ogr …
Wrote FlatGeobuf 11.7 MB → /var/folders/17/gt7fm_5j065bkp6_k04zfrn00000gn/T/tmp2amhqcij.fgb
=== GeoJSON ===
fiona  : 0.939s  (106,504 feat/s)
pyogrio: 1.054s  (94,915 feat/s)

fiona is 1.12x faster (GeoJSON)

=== GeoPackage ===
fiona  : 0.553s  (180,951 feat/s)
pyogrio: 0.416s  (240,482 feat/s)

pyogrio is 1.33x faster (GeoPackage)

=== Shapefile ===
fiona  : 0.654s  (153,016 feat/s)
pyogrio: 0.441s  (226,957 feat/s)

pyogrio is 1.48x faster (Shapefile)

=== FlatGeobuf ===
fiona  : skip
pyogrio: 0.440s  (227,337 feat/s)

As the file size gets larger, pyogrio's advantage grows for most binary formats. For GeoJSON however, the opposite is true; large GeoJSON files get progressively slower with pyogrio. This is a fair tradeoff IMO - the assumption is that if you have large performance-sensitive data, you probably won't be using GeoJSON anyway.

[xjcl]

Thanks a lot for this! 😄 Our team was blocked from migrating to Python 3.14 due to using rasterstats, which requires Fiona, which lacks a compatible binary wheel.

I also find it very amusing that this ticket, which implements Python 3.14 support, has ticket number 314.