validation_io can be simplified

[clessig]

Describe the task. It can be a feature, documentation, etc.

validation_io is currently unnecessarily complex since it converts the new data structures for ModelOutput etc to the old list-of-lists based one to be compatible with weathgen.common.io. The latter should be adapated to directly use the new, dict-based data structures.

CC @grassesi

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

Area

• datasets, data readers, data preparation and transfer
• model
• science
• infrastructure and engineering
• evaluation, export and visualization
• documentation

[tjhunter]

The core issue is OutputBatchData which is based on lists. I would start investigating there, but making sure to time-box this change.

[grassesi]

Yes OutputBatchData should be adapted to work directly with the new output.

[grassesi]

Several issues should be accounted for when implementing this refactor:

• Bug: Inference error when running physical jepa pretrained model #1759
  => Allow for preds to contain 2 tensors one for the latent and the other for the physical values
• Allow to write selected output channels #1705
  => Option to reduce the number of target channels written
• Proper error handling missing when output zarr file exists #1575
  => Guarantee OutputItems are never overwritten (esp. )
• Errors when trying to do inference with fine-tuned JEPA model #1724
  => ignore streams with empty targets