Making GEOARROW geometry encoding work

[Skaidus]

I'm developing a feature that reads parquet files using GEOARROW encoding using duckdb, where GEOARROW_INTERLEAVED is not supported. If I'm not mistaken, this library assumes that GEOARROW_INTERLEAVED is used.

Would there be a way of map to that format before creating the layer in order to be able to use the two?

Thanks in advance.

[kylebarron]

This library should automatically convert the separated coordinate layout into the interleaved layout behind the scenes

[Skaidus]

might be that DuckDB geometry type after a query is not exactly the one that geoarrow exposes?

[kylebarron]

Correct; DuckDB exposes the WKB serialized encoding. We need essentially any of the native encodings in this project.

We need a WKB -> native GeoArrow parser. We started prototyping one but I didn't have the time to finish it. Another route would be to update the Wasm bindings of geoarrow-rs, which has a stable WKB -> GeoArrow implementation.

If you're interested in helping either of those routes let me know. Alternatively you may be able to ask DuckDB to export a GeoArrow native type.

[Skaidus]

Would love to help with the geoarrow-rs bindings, since I think it would make more sense as an arrow feature itself.

Wouldn't also converting from WKB to Geoarrow suffer from some performance overhead? Ideally the binary data that duckDB exposes could be read in the format specified by geoarrow. I don't know if the method register_geoarrow_extensions from this issue you created has something to do with it.

Also thank you for your answer!

[kylebarron]

Each approach has pros and cons. There's an overhead to WebAssembly, so that would be easier to implement but have larger bundle size at runtime.

A pure Typescript implementation would have smallest bundle size but would be most complex to implement correctly.

Wouldn't also converting from WKB to Geoarrow suffer from some performance overhead?

Yes, but that conversion to GeoArrow must happen somewhere since the internal DuckDB geometry format is not GeoArrow.

I don't know if the method register_geoarrow_extensions from this issue you created has something to do with it.

That's about including metadata in the Arrow schema that declares it a geometry column, but that's separate from the type layout of the geometry column.