Feat: launch-slurm should upload important config values when registering an experiment

[tjhunter]

Describe the task. Describe the task. It can be a feature, a set of experiments, documentation, etc.

Use case: as a scientist, I launch a number of experiments such as using this loop:

for lr in "5e-5" "1e-4" "2e-4" "4e-4" ; do
  for node in 2 4 8 ; do
    echo "$lr $node"
    ../WeatherGenerator-private/hpc/launch-slurm.py --chain-jobs 1 --nodes "$node" --options "wgtags.org='ecmwf'" "wgtags.exp='lr_scaling'" "wgtags.issue='1168'" "lr_max=$lr" "num_mini_epochs=1024" "wgtags.num_nodes=$nodes"

  done
done

Currently, the configs and the tags are uploaded at the end of the training run. I need to wait for the completion of the experiment to know the tags associated with the experiment. This prevents me from:

• understanding which run_id is associated with which experiments
• monitoring a large batch of experiments (8+) from within mlflow.

Feature request: when an experiment is launched and registered, also upload the wgtags.* space of the config (at least, maybe also the rest of the config if easy to do).

Marked as initiative because it has to happen after the config is fully resolved.

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

URL to the design document

No response

Area

• datasets, data readers, data preparation and transfer
• model
• science
• infrastructure and engineering
• evaluation, export and visualization
• documentation

[grassesi]

To resolve the config, we can use the machinery from common.config that should be inexpensive to import. Unfortunately we will be missing some information that is not set common.config (eg. stream channel names, etc.), but it will guarantee the same resolution as in the core WeatherGenerator and it will contain most of the information, esp. wgtags.
This can then also act as further motivation to concentrate the rest of the config values that are currently managed outside of common.config. (eg. #954)
With regard to your example above, maybe we should also integrate the capability of running such ablation studies over the parameter space into launch_slurm.py. But probably that is not worth the effort ... (At least for now)

[rolfhm]

As discussed, I can start looking into this