Run inference at 12h steps in both sample and forecast dimension

[iluise]

Describe the task. It can be a feature, documentation, etc.

To save space/time when running inference runs for Quaver we should run inference with the same time steps and frequency as the ones required by the Quaver database:

• initialization times are always at 00:00 and 12:00, so the samples should be distributed with a different frequency (now it's 6h)
• Pangu/GraphCast/AIFS scores have 12h forecast_steps. Is there a way to compute or store only those forecast steps so we can reduce the output zarr by half?

Maybe all these settings already exist and we just need to play with the config. Seeking for advice here and we could open sub-issues for the different tasks that come out of the discussion.

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

Area

• datasets, data readers, data preparation and transfer
• model
• science
• infrastructure and engineering
• evaluation, export and visualization
• documentation

[clessig]

This can for the largest part achieved with:

  
  start_date: ????-??-??T00:00
  end_date: ...

  time_window_step: 12:00:00
  time_window_len: 06:00:00
  
  forecast :
      time_step: 06:00:00
      ...

We then still write 6-hourly forecasts but they have the correct init time (GraphCast is also 6-hourly; we really only store 12-hourly data?).

[ankitpatnala]

If i remember it right when step_hr > len_hr, we get indexing issues

WeatherGenerator/src/weathergen/datasets/multi_stream_data_sampler.py

Line 538 in 0d502ed

step_forecast_dt = base_idx + (self.forecast_delta_dt * fstep) // self.step_timedelta

[clessig]

If i remember it right when step_hr > len_hr, we get indexing issues

WeatherGenerator/src/weathergen/datasets/multi_stream_data_sampler.py

Line 538 in 0d502ed

step_forecast_dt = base_idx + (self.forecast_delta_dt * fstep) // self.step_timedelta

I just tried with

  time_window_step: 12:00:00
  time_window_len: 06:00:00

and the code ran fine (in forecast mode). @ankitpatnala : If there's a problem, could you please open an issue with instructions how to reproduce.

[ankitpatnala]

let me try again with current codebase. There was an issue with output zarr file.
When we (@jehangirawan) did step_hr : 24 and len_hr : 6, forecast_steps : 4, the forecast_step 0,1,2, had the same datetime stamp.

[ankitpatnala]

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=24:00:00

validation_chkpt00000_rank0000.zarr

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')

The target_times for step 1,2, and 3 is same here

[clessig]

training_config.time_window_step=24:00:00

How does the same in the ouput zarr look like for training_config.time_window_step=06:00:00 and training_config.time_window_step=12:00:00 ?

[ankitpatnala]

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=12:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>>

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=06:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T06:00:00', '2022-10-01T06:00:00',
       '2022-10-01T06:00:00', ..., '2022-10-01T06:00:00',
       '2022-10-01T06:00:00', '2022-10-01T06:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T18:00:00', '2022-10-01T18:00:00',
       '2022-10-01T18:00:00', ..., '2022-10-01T18:00:00',
       '2022-10-01T18:00:00', '2022-10-01T18:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')

Even in 12 hour case, the forecast times are not right like in the 6 hour case

[clessig]

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=12:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>>

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=06:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T06:00:00', '2022-10-01T06:00:00',
       '2022-10-01T06:00:00', ..., '2022-10-01T06:00:00',
       '2022-10-01T06:00:00', '2022-10-01T06:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T18:00:00', '2022-10-01T18:00:00',
       '2022-10-01T18:00:00', ..., '2022-10-01T18:00:00',
       '2022-10-01T18:00:00', '2022-10-01T18:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')

Even in 12 hour case, the forecast times are not right like in the 6 hour case

Thanks. Yes, the 12h case is also not what I expected.

[ankitpatnala]

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=12:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>>

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=06:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T06:00:00', '2022-10-01T06:00:00',
       '2022-10-01T06:00:00', ..., '2022-10-01T06:00:00',
       '2022-10-01T06:00:00', '2022-10-01T06:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T18:00:00', '2022-10-01T18:00:00',
       '2022-10-01T18:00:00', ..., '2022-10-01T18:00:00',
       '2022-10-01T18:00:00', '2022-10-01T18:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')

Even in 12 hour case, the forecast times are not right like in the 6 hour case

Thanks. Yes, the 12h case is also not what I expected.

Do we intend to use this configuration ever?

[clessig]

Yes, this is high-priority to be fixed.