Forecast steps time index error when step_hr exceeds len_hr length

[ankitpatnala]

What happened?

When the configuration of step_hr is more than `len_hr', the forecast steps behaves not in a right manner. There are indexing issues

WeatherGenerator/src/weathergen/datasets/multi_stream_data_sampler.py

Line 538 in 0d502ed

step_forecast_dt = base_idx + (self.forecast_delta_dt * fstep) // self.step_timedelta

for settings when step_hr =12:00:00 and len_hr=06:00:00
self.forecast_dt = 06:00:00
self.step_time_delta = 12:00:00

for fstep = 1, it becomes 6*1 // 12 which equals 0. So, there is no increment of step_forecast_dt

It can be seen in this zarr file

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=24:00:00

validation_chkpt00000_rank0000.zarr

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')

The target_times for step 1,2, and 3 is same here

inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=12:00:00

>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
       '2022-10-01T00:00:00', '2022-10-01T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
       '2022-10-01T12:00:00', '2022-10-01T12:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
       '2022-10-02T00:00:00', '2022-10-02T00:00:00'],
      shape=(40320,), dtype='datetime64[s]')
>>>

This is also addressed in Issue #1595

What are the steps to reproduce the bug?

To reproduce: train a sample model with default config and use the inference command as stated above. Inspect the zarr file afterwards.

Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.

No response

[ankitpatnala]

assign me

[github-actions]

Issue assigned to @ankitpatnala.

[grassesi]

Please also coordinate with @enssow who is working on #1438 , As both issues probably interact.