grid_cell_polygons_from_cellids failed at high resolutions

[thangqd]

When calling grid_cell_polygons_from_cellids for ISEA3H at resolutions <= 21, it works fine

dggrid_instance.grid_cell_polygons_from_cellids( ['84559272651'], 'ISEA3H', 21, split_dateline=True ) ,

When calling grid_cell_polygons_from_cellids for ISEA3H at resolutions > 21 , got error:

dggrid_instance.grid_cell_polygons_from_cellids(['253677640803'], 'ISEA3H', 22, split_dateline=True )`

THE SAME ISSUE OCCURED for other DGGS_TYPES

---

ValueError Traceback (most recent call last)
/tmp/ipython-input-100303072.py in <cell line: 0>()
1 from vgrid.conversion.dggs2geo.dggrid2geo import dggrid2geo
----> 2 dggrid_geo = dggrid2geo(dggrid_instance, dggs_type,dggrid_id,resolution, address_type)
3 dggrid_geo

2 frames
/usr/local/lib/python3.12/dist-packages/dggrid4py/dggrid_runner.py in dgapi_grid_gen(self, dggs, subset_conf, output_conf)
702 if self.capture_logs == True:
703 message = f"some error happened under the hood of dggrid (exit code {result}): " + self.last_run_logs
--> 704 raise ValueError(message)
705 else:
706 message = f"some error happened under the hood of dggrid (exit code {result}), try capture_logs=True for dggrid instance"

ValueError: some error happened under the hood of dggrid (exit code 1): ** executing DGGRID version 8.33 without GDAL **
type sizes: big int: 64 bits / big double: 128 bits

** using meta file metafile_7b27444c-4d5e-4dbb-991c-53e81055d4fa...

• parameter values:
  dggrid_operation GENERATE_GRID (user set)
  rng_type RAND (default)
  precision 7 (default)
  verbosity 0 (default)
  pause_on_startup false (default)
  pause_before_exit false (default)
  update_frequency 100000 (default)
  dggs_type ISEA3H (user set)
  dggs_topology HEXAGON (user set)
  dggs_proj ISEA (user set)
  dggs_aperture_type PURE (user set)
  dggs_aperture 3 (user set)
  proj_datum WGS84_AUTHALIC_SPHERE (default)
  dggs_orient_specify_type SPECIFIED (user set)
  dggs_num_placements 1 (user set)
  dggs_vert0_lon 11.25 (user set)
  dggs_vert0_lat 58.2825 (user set)
  dggs_vert0_azimuth 0 (user set)
  dggs_res_specify_type SPECIFIED (user set)
  dggs_res_spec 22 (user set)
  input_files vals.txt (default)
  input_file_name valsin.txt (default)
  point_input_file_type NONE (default)
  input_address_type GEO (default)
  input_delimiter " " (default)
  output_file_name valsout.txt (default)
  output_file_type NONE (default)
  output_address_type SEQNUM (user set)
  output_delimiter " " (default)
  densification 0 (default)
  longitude_wrap_mode WRAP (default)
  unwrap_points true (default)
  output_cell_label_type GLOBAL_SEQUENCE (default)
  cell_output_type SHAPEFILE (user set)
  point_output_type NONE (user set)
  randpts_output_type NONE (default)
  cell_output_file_name /content/temp_ISEA3H_22_out_ad625e65-d083-4fd5-8f15-281df85af924 (user set)
  point_output_file_name centers (default)
  randpts_output_file_name randPts (default)
  collection_output_file_name cells (default)
  dggs_orient_output_file_name grid.meta (default)
  shapefile_id_field_length 11 (default)
  kml_default_width 4 (default)
  kml_default_color ffffffff (default)
  kml_name (default)
  kml_description Generated by DGGRID 8.33 (default)
  neighbor_output_type NONE (default)
  neighbor_output_file_name nbr (default)
  children_output_type NONE (default)
  children_output_file_name chld (default)
  max_cells_per_output_file 0 (default)
  output_first_seqnum 1 (default)
  output_last_seqnum 18446744073709551615 (default)
  geodetic_densify 0 (default)
  clip_subset_type SEQNUMS (user set)
  clip_cell_addresses (default)
  clip_cell_res 1 (default)
  clip_cell_densification 1 (default)
  clip_region_files /content/temp_clip_ad625e65-d083-4fd5-8f15-281df85af924.txt (user set)
  clip_type POLY_INTERSECT (default)
  clipper_scale_factor 1000000 (default)

FATAL ERROR: DgOutShapefile::writeDbf() index string length of 12 exceeds value of parameter shapefile_id_field_length.

[allixender]

Hi. That's a "good one" and points to some challenges with the shapefile intermediate output. The last log row is helpful FATAL ERROR: DgOutShapefile::writeDbf() index string length of 12 exceeds value of parameter shapefile_id_field_length.

In the portable DGGRID version, the only useful intermediate output formats would be shapefile, and maybe GeoJSON. It will not happen with GeoPackage or FlatGeoBuf, but these GDAL formats are not available in the protable version. I am not sure where this exactly breaks, because I am only aware that the field names in shapefile cannot be longer than 10 characters (classic annoyance). But this seems to be related to the data in the field? Anyway, this is both an upstream DGGRID issue, and/or portable compile issue.

We are working on adding native GeoArrow (feather and IPC file format) to a portable version, that might eventually allow us to leave "legacy" exchange formats behind.

It seems DGGRID itself support a GeoJSON driver (independent of GDAL). I could try to add an additional config option to enable this GEOJSON (https://dggrid.readthedocs.io/latest/dggrid_man_V841.html#specifying-per-cell-grid-output) driver as a legacy format (https://github.com/allixender/dggrid4py/blob/master/dggrid4py/dggrid_runner.py#L32) when instantiating the DGGRIDv7 runner (https://dggrid4py.readthedocs.io/en/stable/dggrid4py/dggrid4py.DGGRIDv7.html#dggrid4py.DGGRIDv7.__init__) ... ?

TODO 👍

[tik65536]

I also encountered this issue when I was testing on issue #19 with parameter clip_subset_type=CORASE_CELLS and resolution >= 10 . The length of the cell ID is greater than 11.

I found that one of the DGGRID meta file parameters may help to resolve this case: shapefile_id_field_length. You can find the setting in the appendix A. DGGRID Metafile Parameters of the DGGRID documentation v 8.41.

Hope this helps.
Tik

[allixender]

We will look into making this work. It will take some time.

[tik65536]

Hey, @allixender,

I had tried to add the field shapefile_id_field_length 50 to the meta file with refinement level 12. It works.
So you can set it to the maximum value of 50 to fit arbitrary refinement levels.

I have attached the meta file, shape files and the output here for reference.

metafile_465d724b-5d96-4544-ace6-5f323328f6f8.txt

temp_clip_aa5a663e-1d77-4cb4-9abc-df02d0690a86.zip

[allixender]

Should be included into next release 👍 TODO 😁

[fmigneault]

Looking forward to the new release. I just encountered it with grid_cell_polygons_for_extent using resolution=15 with ISEA7h/IGEO7.

[allixender]

Config parameter should be doable in DGGRIDv8 wrapper class, and maybe in DGGRIDv7 backported but only if the legacy param is set to True?