Sorcha/dev/zarr3 compaction

[enssow]

Description

Draft PR to showcase how to enable sharding (experimentation underway to optimise performance)

Issue Number

DRAFT
Closes #1384

Checklist before asking for review

• I have performed a self-review of my code
• My changes comply with basic sanity checks:
  • I have fixed formatting issues with ./scripts/actions.sh lint
  • I have run unit tests with ./scripts/actions.sh unit-test
  • I have documented my code and I have updated the docstrings.
  • I have added unit tests, if relevant
• I have tried my changes with data and code:
  • I have run the integration tests with ./scripts/actions.sh integration-test
  • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
  • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
• I have informed and aligned with people impacted by my change:
  • for config changes: the MatterMost channels and/or a design doc
  • for changes of dependencies: the MatterMost software development channel

[grassesi]

I think it would be better if zarr_store would be under general.zarr_store but this is just nitpicking at this point. Please do remove the print statements tho.

[enssow]

I think it would be better if zarr_store would be under general.zarr_store but this is just nitpicking at this point. Please do remove the print statements tho.

Sorry, think the commit was not the most recent! prints all gone :) i did think about putting it under a different category but the default_forecast_config doesn't have a general section so it may be best just where it is - let me know what you think?

[enssow]

type-check fails due to an import error with tqdm (this can be resolved by directly adding tqdm to the dependencies of common pyproject.toml. However, this then leads to further issues:

ERROR Type `object` is not iterable [not-iterable]
   --> packages/common/src/weathergen/common/io.py:412:24
    |
412 |         for dataset in tqdm(item.datasets):
    |                        ^^^^^^^^^^^^^^^^^^^
    |
 INFO 1 error (42 ignored)

Which is a known issue with pryefly
To discuss what is the best resolution :)