Verify HSP* and Update RegressTest to run under Pandas 3.0.0

[rburghol]

Relevant to PRs #216 which achieved preliminary successful model runs under pandas 3.0.0, and eliminates deprecated pandas methods and argument type restrictions.

• tests/convert/regression_base.py under pandas 3.0.0 fails with an error pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).
  • SEDTRN validation looks at the VOL (HYDR) output and zeroes does with ultra low reach water.
  • But the HSPF and hsp2 VOL timeseries are misaligned See https://github.com/respec/HSPsquared/actions/runs/21381288537/job/61548511129?pr=207
• RegressTest class imports HDF5 from src/hsp2/hsp2tools/HDF5.py whereas main.py imports HDF5 from src/hsp2hsp2io/hdf.py
  • Need to explore whether the hsp2io:HDF5:read_ts() and hsp2tools:HDF5:get_time_series() behave the same way
  • Need to understand if these need to be converged (recall a previous planning group meeting indicated this as important). @timcera
• Verify: Closing the hdf5 at the end of the run eliminates the pytest error: ValueError: The file '/opt/model/HSPsquared/tests/test10/HSPFresults/test10.h5' is already opened, but not in read-only mode (as requested). in pandas >= 3.0.0
• Why does HSP2 VOL timeseries have the misaligned timestamp? (see Table 1)

Misaligned SEDTRN timeseries produces pandas 3.0.0 test error.

• The error that is thrown in pandas-3.0.0 tests -- either: pandas-2.3.3 does not produce this misalignment, or the indices have shifted from a simple sequential integer to a timestamp.
• The time index im the hsp2 series is really wacked out
• The test code that fails looks to find hsp2 timesteps where the flow volume is < 1.0e-4, and then sets the corresponding SSED values to 0.0. However, then it goes on to also try to set the same timeslots to 0.0 in the HSPF SSED timeseries. This causes an error because the timestamp indices are misaligned.

            idx_low_vol = ts_vol_hsp2 < 1.0e-4
            ts_hsp2.loc[idx_low_vol] = ts_hsp2.loc[idx_low_vol] = 0
            ts_hspf.loc[idx_low_vol] = ts_hspf.loc[idx_low_vol] = 0

• The fact that the test TRIES this is an error IMO because I think we should not be trying to zero the HSPF SSED on days when the hsp2 has miniscule flow since this could paper over actual differences in the SSED simulation. I think it would be correct to set HSPF SSED to 0.0 when HSPF volume is < 1.0e-4.

Table 1: Timeseries output from hsp2 versus hspf showing a timestamp disagreement.

>>> ts_hsp2
index
1970-01-03 04:35:06.000    29.916803
1970-01-03 04:35:09.600    29.834137
1970-01-03 04:35:13.200    29.751753
                             ...
1970-01-03 13:21:57.600     0.000000
1970-01-03 13:22:01.200     0.000000
1970-01-03 13:22:04.800     0.000000
Name: SSED3, Length: 8784, dtype: float32
>>> ts_hspf
1976-01-01 01:00:00    29.916805
1976-01-01 02:00:00    29.834139
1976-01-01 03:00:00    29.751757
                         ...
1976-12-31 22:00:00     0.000000
1976-12-31 23:00:00     0.000000
1977-01-01 00:00:00     0.000000
Name: SSEDCLAY, Length: 8784, dtype: float64

Table 2: Excerpt of error text from https://github.com/respec/HSPsquared/actions/runs/21381288537/job/61548511129?pr=207

tests/test_regression.py:49: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/convert/regression_base.py:225: in check_con
    ts_hsp2, ts_hspf = self.validate_time_series(
tests/convert/regression_base.py:329: in validate_time_series
    ts_hspf.loc[idx_low_vol] = ts_hspf.loc[idx_low_vol] = 0
    ^^^^^^^^^^^^^^^^^^^^^^^^

...
E               pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/site-packages/pandas/core/indexing.py:2677: IndexingError

[timcera]

You need to edit pyproject.toml to have "pandas<3.0" instead of just "pandas". The pyproject.toml is used by pip to install and also to prepare a package for upload to pypi.org. The environment*.yml files are used by conda and also need to be edited.

[rburghol]

Thanks @timcera !!!

[rburghol]

@timcera I think I recall you raising the issue about the 2 HDF5 classes and how that could be a problem, and I think this issue raises the possibility that this is in fact causing a divergence. I think this shows the difference in timeseries indices is the result of the 2 different HDF5 classes, one of which is used in regression_test.py and the other which is used by main.py.:

# Note: This code must be run from the tests dir due to importing
# the `convert` directory where the regression_base file lives
# todo: make this convert path passable by argument 
import os

import numpy
from hsp2.hsp2.main import *
from hsp2.hsp2.om import *
from hsp2.hsp2io.hdf import HDF5
from hsp2.hsp2io.io import IOManager
from hsp2.state.state import *

import numpy as np
from convert.regression_base import RegressTest

case = "test10specl"
base_case = "test10"
#case = "testcbp"
tdir = "/opt/model/HSPsquared/tests"
#import_uci(str(hsp2_specl_uci), str(temp_specl_h5file))
#run(temp_specl_h5file, saveall=True, compress=False)
fpath = "./test10specl/HSP2results/test10specl.h5"
# try also:
# fpath = './tests/testcbp/HSP2results/JL1_6562_6560.h5'
# sometimes when testing you may need to close the file, so try:
# f = h5py.File(fpath,'a') # use mode 'a' which allows read, write, modify
# # f.close()
hdf5_instance = HDF5(fpath)
io_manager = IOManager(hdf5_instance)

############# RegressTest data loader
## TEST CASE
test = RegressTest(case, threads=1)

params = ('RCHRES', 'SEDTRN', '001', 'SSEDCLAY', '2')
test.check_con(params = params)
operation, activity, id, constituent, tcode = params
ts_hsp2 = test.hsp2_data.get_time_series(operation, id, constituent, activity)
ts_hspf = test.get_hspf_time_series(params)

# begin checking code for validate_time_series()
if len(ts_hsp2) > len(ts_hspf):
    ts_hsp2 = ts_hsp2[0 : len(ts_hspf)]

ts_hsp2 = test.fill_nan_and_null(ts_hsp2)
ts_hspf = test.fill_nan_and_null(ts_hspf)

### special cases
# SEDTRN
ts_vol_hsp2 = test.hsp2_data.get_time_series(operation, id, "VOL", "HYDR")
ts_vol_hsp2 = test.fill_nan_and_null(ts_vol_hsp2)

idx_low_vol = ts_vol_hsp2 < 1.0e-4
ts_hsp2.loc[idx_low_vol] = ts_hsp2.loc[idx_low_vol] = 0
ts_hspf.loc[idx_low_vol] = ts_hspf.loc[idx_low_vol] = 0


siminfo["tindex"] = date_range(siminfo["start"], siminfo["stop"], freq=Minute(siminfo["delt"]))[1:]
siminfo["steps"] = len(siminfo["tindex"])

params = ('RCHRES', 'SEDTRN', '001', 'SSEDCLAY', '2')
operation, activity, id, constituent, tcode = params

params = ('RCHRES', 'HYDR', '001', 'IVOL', '2')
segment = operation[0] + id 
operation, activity, id, constituent, tcode = params

tsivol_io = io_manager.read_ts(Category.RESULTS, operation, segment, activity)
tsivol_tools = test.hsp2_data.get_time_series(operation, id, "IVOL", activity)
np.quantile(tsivol_io.IVOL, [0,0.25,0.5,0.75,1.0])
np.quantile(tsivol_tools, [0,0.25,0.5,0.75,1.0])
tsivol_io.index
tsivol_tools.index

Output:

• quantiles match (as do values in order)
• indices are misaligned -- hsp2.hsp2tools.HDF5 has decimal hours

array([0.        , 0.        , 0.0644508 , 0.49963082, 2.85344005])
>>> np.quantile(tsivol_tools, [0,0.25,0.5,0.75,1.0])
array([0.        , 0.        , 0.0644508 , 0.49963082, 2.85344005])
>>> tsivol_io.index
DatetimeIndex(['1976-01-01 01:00:00', '1976-01-01 02:00:00',
               '1976-01-01 03:00:00', '1976-01-01 04:00:00',
               '1976-01-01 05:00:00', '1976-01-01 06:00:00',
               '1976-01-01 07:00:00', '1976-01-01 08:00:00',
               '1976-01-01 09:00:00', '1976-01-01 10:00:00',
               ...
               '1976-12-31 15:00:00', '1976-12-31 16:00:00',
               '1976-12-31 17:00:00', '1976-12-31 18:00:00',
               '1976-12-31 19:00:00', '1976-12-31 20:00:00',
               '1976-12-31 21:00:00', '1976-12-31 22:00:00',
               '1976-12-31 23:00:00', '1977-01-01 00:00:00'],
              dtype='datetime64[us]', length=8784, freq='60min')
>>> tsivol_tools.index
DatetimeIndex([       '1970-01-03 04:35:06', '1970-01-03 04:35:09.600000',
               '1970-01-03 04:35:13.200000', '1970-01-03 04:35:16.800000',
               '1970-01-03 04:35:20.400000',        '1970-01-03 04:35:24',
               '1970-01-03 04:35:27.600000', '1970-01-03 04:35:31.200000',
               '1970-01-03 04:35:34.800000', '1970-01-03 04:35:38.400000',
               ...
               '1970-01-03 13:21:32.400000',        '1970-01-03 13:21:36',
               '1970-01-03 13:21:39.600000', '1970-01-03 13:21:43.200000',
               '1970-01-03 13:21:46.800000', '1970-01-03 13:21:50.400000',
                      '1970-01-03 13:21:54', '1970-01-03 13:21:57.600000',
               '1970-01-03 13:22:01.200000', '1970-01-03 13:22:04.800000'],
              dtype='datetime64[ns]', name='index', length=8784, freq=None)