Add STAC <> Zarr report #139
Conversation
maxrjones
left a comment
maxrjones
left a comment
There was a problem hiding this comment.
This is wonderful, thank you @jsignell!
This is a tremendous resource as-is! I think it could be taken to the next level, separately from this PR, by having a small demonstration of catalogs that employ each of these approaches perhaps with a STAC browser on top. Do you have any plans to build of resource of that type? I'd be glad to help.
| - Store the result of a data-cube constructed by concatenating Zarr stores: | ||
| - as a new Zarr store - this option can include filtering and subsetting | ||
| - as a virtual reference file (icechunk or kerchunk) |
There was a problem hiding this comment.
@mdsumner have you tried out concatenating Zarr stores via VRT/GTI? I'm struggling to keep up with your work, but thought you may have shared this as an option as well.
There was a problem hiding this comment.
Right! I had seen https://www.hypertidy.org/posts/2025-03-12-r-py-multidim/r-py-multidim but I need to revisit and link out.
Actually, I'm now wondering if even adding some drawing to these pages would help clarify the concepts as much or more than a full demonstration. |
|
I also wonder if the virtual zarr section should include some guidance on how to organize cataloging of the raw data with cataloging of the virtual Zarr references/stores? |
Yes! I think adding drawings could be really helpful. I was struggling with how to structure the sections, but once we settle on that drawings would really be useful. I think it would also help to just point to examples that implement the different setups. |
Yeeeaaahhh I wasn't at sure how much to talk about virtual zarr... but I can imagine a scenario where you catalog the data in normal stac objects but then also include a top-level reference file/store so that you have best of both worlds in terms of data access, but then you have 2 ways of accessing data and 2 places where you are abstracting metadata so I wasn't sure if that is a good idea. |
Maybe a good balance would be just to open an issue for a TODO on additional guidance for cataloguing both raw data and virtual zarrs and mention that issue in this new section, so readers are aware it's not fully comprehensive. |
|
Ok I did the text changes and added links to data-consumers at least. I'm going to work on some drawings next. |
|
Ok I made these diagrams:
|
gadomski
left a comment
gadomski
left a comment
There was a problem hiding this comment.
Sorry it took me so long to get around to another read-through. I really like it ... feels like all the "big ideas" for STAC (i.e. STAC isn't for data, STAC is good at L2 and below, etc) are covered.
I did a read-through mostly with my "experienced practitioner" hat on, so I don't have much feedback yet on how it might read to someone newer to the space. Depending on when this is intended to ship, I can try to take another pass next week from a "newcomer" perspecitve (if I even can get to that perspective 😬).
Co-authored-by: Pete Gadomski <[email protected]>
|
Ok I am going to try to make this build work and then I feel like this is in a pretty good spot! |
|
My understanding is that this fork needs to be merged to staging before the preview action will work. |
|
@jsignell yes only branches on the primary fork work with preview, we should actually disable attempts to preview remote forks. |
4 participants
First draft of #134 borrowing heavily from conversation on that issue.