make `update_uci` function to omit WDM updates

- https://github.com/HARPgroup/HSPsquared/blob/develop-state-class/tests/testcbp/HSP2results/PL3_5250_0001eq.uci

```
  File "/usr/local/share/venv/hsp2dev_py10/lib/python3.10/site-packages/hsp2/hsp2io/io.py", line 72, in read_parameters
    return self._parameters.read_parameters()
  File "/usr/local/share/venv/hsp2dev_py10/lib/python3.10/site-packages/hsp2/hsp2io/hdf.py", line 73, in read_parameters
    for id, vdict in self._store[path].to_dict("index").items():

...

TypeError: cannot properly create the storer for: [_TABLE_MAP] [group->/PERLND/SNOW/STATES (Group) '',value-><class 'NoneType'>,format->None

```

#### Case Study
- Can import original UCI and run 1 year 2001-2002
- Can update_uci and run 40 years
- 3rd update causes error `TypeError: cannot properly create the storer for: [_TABLE_MAP] [group->/PERLND/SNOW/STATES (Group) '',value-><class 'NoneType'>,format->None`
- Questions:
   - Is the 2nd step (update_uci) actually updating all tables, or rather, preserving them?

##### Load the Libs and hdf5
```
import os
import numpy
import h5py
from hsp2.hsp2.main import *
from hsp2.state.state import *
from hsp2.hsp2.om import *
from hsp2.hsp2.SPECL import *
from hsp2.hsp2io.hdf import HDF5
from hsp2.hsp2io.io import IOManager
from hsp2.hsp2tools.readUCI import *
from src.hsp2.hsp2tools.commands import import_uci, run
from pandas import read_hdf

fpath = "./tests/testcbp/HSP2results/PL3_5250_0001eq.h5"
# to run use this:
# run(fpath, saveall=True, compress=False)
store = pd.HDFStore(str(fpath), mode='r')

```

##### Explore the dictionary
- Find stuff that breaks things line 72 in `hdf.py`
- verify contents across imports
- Shows the keys: `dstore_hydreq.keys()`
- The error occurs when the loop reaches `/PERLND/SNOW/STATES`, in the command: `dstore_hydreq['/PERLND/SNOW/STATES'].keys()`

##### The test 
- 1st `import_uci`, `dstore_hydreq['/PERLND/SNOW/STATES'].keys()`:
```
Index(['OPNID', 'PKSNOW', 'PKICE', 'PKWATR', 'RDENPF', 'DULL', 'PAKTMP',
       'COVINX', 'XLNMLT', 'SKYCLR'],
      dtype='object')
```
- Step 2: `hsp2 update_uci PL3_5250_0001eqlong.uci PL3_5250_0001eq.h5`, `dstore_hydreq['/PERLND/SNOW/STATES'].keys()`:
```
Index(['OPNID', 'PACKF', 'PACKI', 'PACKW', 'RDENPF', 'DULL', 'PAKTMP',
       'COVINX', 'XLNMLT', 'SKYCLR', 'PKSNOW', 'PKICE', 'PKWATR'],
      dtype='object')
```
- Step 3: do it again (don't try to go back to original UCI) `hsp2 update_uci PL3_5250_0001eqlong.uci PL3_5250_0001eq.h5`, 
```
 readUCI(ucifile, h5file, False)
  File "/usr/local/share/venv/hsp2dev_py10/lib/python3.10/site-packages/hsp2/hsp2tools/readUCI.py", line 318, in readUCI

```

##### Debugging Code
```

# this is the code from hdf.py that breaks
for path in dstore_hydreq.keys():
    op, module, *other = path[1:].split(sep="/", maxsplit=3)
    print(path, op, module)

dstore_hydreq.close()

path = "/PERLND/SNOW/STATES"
pd.read_hdf(dstore_hydreq, path)
>      OPNID  PKSNOW  PKICE  PKWATR  RDENPF   DULL  PAKTMP  COVINX  XLNMLT  SKYCLR
> R001   NaN     0.0    0.0     0.0     0.2  400.0    32.0    0.01     0.0     1.0

# The OPNID is NaN is this a problem?

# Probably not since the previous dataframe ALSO has a NaN OPNID and it was fine
path="/RCHRES/ACIDPH/STATES"
     OPNID  ACCONC1  ACCONC2  ACCONC3  ACCONC4  ACCONC5  ACCONC6  ACCONC7
R001   NaN      0.0      0.0      0.0      0.0      0.0      0.0      0.0
```
- Now try to do the code in question and then re-read and see if we can get an error
   - start by re-storing the original h5: `cp PL3_5250_0001eq.h5.orig PL3_5250_0001eq.h5`
   - Run Lines 312-315 in https://github.com/HARPgroup/HSPsquared/blob/develop-state-class/src/hsp2/hsp2tools/readUCI.py
```
fpath = "./tests/testcbp/HSP2results/PL3_5250_0001eq.h5"
store = pd.HDFStore(str(fpath), mode='a')
path = "/PERLND/SNOW/STATES"
df = pd.read_hdf(store, path)
df = df.rename(
    columns={"PKSNOW": "PACKF", "PKICE": "PACKI", "PKWATR": "PACKW"}
)
df.to_hdf(store, key=path, data_columns=True)
store.close()
```
- Then repeat the above same code again as if rerunning from CMD line
   - This DOES NOT CAUSE AN ERROR
   - So maybe it is later when defaults are applied?
- Create the defaults frame:
```
parse = defaultdict(list)
defaults = {}
cat = {}
path = {}
hsp_paths = {}
hsp_paths = {}
datapath = os.path.join(hsp2tools.__path__[0], "data", "ParseTable.csv")
for row in pd.read_csv(datapath).itertuples():
    parse[row.OP, row.TABLE].append(
        (row.NAME, row.TYPE, row.START, row.STOP, row.DEFAULT)
    )
    defaults[row.OP, row.SAVE, row.NAME] = convert[row.TYPE](row.DEFAULT)
    cat[row.OP, row.TABLE] = row.CAT
    path[row.OP, row.TABLE] = row.SAVE
    # store paths for checking defaults:
    hsp_path = f"/{row.OP}/{row.SAVE}/{row.CAT}"
    if not hsp_path in hsp_paths:
        hsp_paths[hsp_path] = {}
    hsp_paths[hsp_path][row.NAME] = defaults[row.OP, row.SAVE, row.NAME]


```
- Now apply defaults for "/PERLND/SNOW/STATES"
```
path = "/PERLND/SNOW/STATES"
df = pd.read_hdf(store, path)
dct_params = hsp_paths[path]
new_columns = {}
for par_name in dct_params:
    if par_name == "CFOREA":
        ichk = 0
    if par_name not in df.columns:  # missing value in HDF5 path
        def_val = dct_params[par_name]
        if def_val != "None":
            # df[par_name] = def_val
            new_columns[par_name] = def_val

new_columns = pd.DataFrame(new_columns, index=df.index)
# now this adds duplicate columns, that are going to cause us problems perhaps when we rename them to existing columns?
df1 = pd.concat([df, new_columns], axis=1)

```

### The Entire Code as command line runnable test
```
"""
Read data from a UCI file and create an HDF file with the data.

Parameters
----------
uciname : str
    The name of the UCI file to read.
hdfname : str
    The name of the HDF file to store the data.
overwrite : bool, optional
    Whether to overwrite existing data in the HDF file. Defaults to True.

Returns
-------
None
"""
# Hard code inputs for example testing
uciname = "./tests/testcbp/HSP2results/PL3_5250_0001eq.uci"
hdfname = "./tests/testcbp/HSP2results/PL3_5250_0001eq.h5"
overwrite = False


if overwrite is True and os.path.exists(hdfname):
    os.remove(hdfname)

# create lookup dictionaries from 'ParseTable.csv' and 'rename.csv'
parse = defaultdict(list)
defaults = {}
cat = {}
path = {}
hsp_paths = {}
datapath = os.path.join(hsp2tools.__path__[0], "data", "ParseTable.csv")
for row in pd.read_csv(datapath).itertuples():
    parse[row.OP, row.TABLE].append(
        (row.NAME, row.TYPE, row.START, row.STOP, row.DEFAULT)
    )
    defaults[row.OP, row.SAVE, row.NAME] = convert[row.TYPE](row.DEFAULT)
    cat[row.OP, row.TABLE] = row.CAT
    path[row.OP, row.TABLE] = row.SAVE
    # store paths for checking defaults:
    hsp_path = f"/{row.OP}/{row.SAVE}/{row.CAT}"
    if not hsp_path in hsp_paths:
        hsp_paths[hsp_path] = {}
    hsp_paths[hsp_path][row.NAME] = defaults[row.OP, row.SAVE, row.NAME]

rename = {}
extendlen = {}
datapath = os.path.join(hsp2tools.__path__[0], "data", "rename.csv")
for row in pd.read_csv(datapath).itertuples():
    if row.LENGTH != 1:
        extendlen[row.OPERATION, row.TABLE] = row.LENGTH
    rename[row.OPERATION, row.TABLE] = row.RENAME

net = None
sc = None
store = pd.HDFStore(hdfname, mode="a")
info = (store, parse, path, defaults, cat, rename, extendlen)
f = reader(uciname)
for line in f:
    if line.startswith("GLOBAL"):
        global_(info, getlines(f))
    elif line.startswith("OPN"):
        opn(info, getlines(f))
    elif line.startswith("NETWORK"):
        net = network(info, getlines(f))
    elif line.startswith("SCHEMATIC"):
        sc = schematic(info, getlines(f))
    elif line.startswith("MASS-LINK"):
        masslink(info, getlines(f))
    elif line.startswith("FTABLES"):
        ftables(info, getlines(f))
    elif line.startswith("EXT"):
        ext(info, getlines(f))
    elif line.startswith("GENER"):
        gener(info, getlines(f))
    elif line.startswith("PERLND"):
        operation(info, getlines(f), "PERLND")
    elif line.startswith("IMPLND"):
        operation(info, getlines(f), "IMPLND")
    elif line.startswith("RCHRES"):
        operation(info, getlines(f), "RCHRES")
    elif line.startswith("MONTH-DATA"):
        monthdata(info, getlines(f))
    elif line.startswith("SPEC-ACTIONS"):
        specactions(info, getlines(f))

colnames = (
    "AFACTR",
    "MFACTOR",
    "MLNO",
    "SGRPN",
    "SMEMN",
    "SMEMSB",
    "SVOL",
    "SVOLNO",
    "TGRPN",
    "TMEMN",
    "TMEMSB",
    "TRAN",
    "TVOL",
    "TVOLNO",
    "COMMENTS",
)
if not ((net is None) and (sc is None)):
    linkage = pd.concat((net, sc), ignore_index=True, sort=True)
    for cname in colnames:
        if cname not in linkage.columns:
            linkage[cname] = ""
    linkage = linkage.sort_values(by=["TVOLNO"]).replace("na", "")
    linkage.to_hdf(store, key="/CONTROL/LINKS", data_columns=True)

Lapse.to_hdf(store, key="TIMESERIES/LAPSE_Table")
Seasons.to_hdf(store, key="TIMESERIES/SEASONS_Table")
Svp.to_hdf(store, key="TIMESERIES/Saturated_Vapor_Pressure_Table")
keys = set(store.keys())
# rename needed for restart. NOTE issue with line 157 in PERLND SNOW HSPF
# where PKSNOW = PKSNOW + PKICE at start - ONLY
path = "/PERLND/SNOW/STATES"
if path in keys:
    df = pd.read_hdf(store, path)
    df = df.rename(
        columns={"PKSNOW": "PACKF", "PKICE": "PACKI", "PKWATR": "PACKW"}
    )
    df.to_hdf(store, key=path, data_columns=True)

path = "/IMPLND/SNOW/STATES"
if path in keys:
    df = pd.read_hdf(store, path)
    df = df.rename(
        columns={"PKSNOW": "PACKF", "PKICE": "PACKI", "PKWATR": "PACKW"}
    )
    df.to_hdf(store, key=path, data_columns=True)

path = "/PERLND/SNOW/FLAGS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "SNOPFG" not in df.columns:  # didn't read SNOW-FLAGS table
        df["SNOPFG"] = 0
        df.to_hdf(store, key=path, data_columns=True)

path = "/IMPLND/SNOW/FLAGS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "SNOPFG" not in df.columns:  # didn't read SNOW-FLAGS table
        df["SNOPFG"] = 0
        df.to_hdf(store, key=path, data_columns=True)

# Need to fixup missing data
path = "/IMPLND/IWATER/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "PETMIN" not in df.columns:  # didn't read IWAT-PARM2 table
        df["PETMIN"] = 0.35
        df["PETMAX"] = 40.0
        df.to_hdf(store, key=path, data_columns=True)

path = "/IMPLND/IWTGAS/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "SDLFAC" not in df.columns:  # didn't read LAT-FACTOR table
        df["SDLFAC"] = 0.0
        df["SLIFAC"] = 0.0
        df.to_hdf(store, key=path, data_columns=True)

path = "/IMPLND/IQUAL/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "SDLFAC" not in df.columns:  # didn't read LAT-FACTOR table
        df["SDLFAC"] = 0.0
        df["SLIFAC"] = 0.0
        df.to_hdf(store, key=path, data_columns=True)

path = "/PERLND/PWTGAS/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "SDLFAC" not in df.columns:  # didn't read LAT-FACTOR table
        df["SDLFAC"] = 0.0
        df["SLIFAC"] = 0.0
        df["ILIFAC"] = 0.0
        df["ALIFAC"] = 0.0
        df.to_hdf(store, key=path, data_columns=True)
    if "SOTMP" not in df.columns:  # didn't read PWT-TEMPS table
        df["SOTMP"] = 60.0
        df["IOTMP"] = 60.0
        df["AOTMP"] = 60.0
        df.to_hdf(store, key=path, data_columns=True)
    if "SODOX" not in df.columns:  # didn't read PWT-GASES table
        df["SODOX"] = 0.0
        df["SOCO2"] = 0.0
        df["IODOX"] = 0.0
        df["IOCO2"] = 0.0
        df["AODOX"] = 0.0
        df["AOCO2"] = 0.0
        df.to_hdf(store, key=path, data_columns=True)

path = "/PERLND/PWATER/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "FZG" not in df.columns:  # didn't read PWAT-PARM5 table
        df["FZG"] = 1.0
        df["FZGL"] = 0.1
        df.to_hdf(store, key=path, data_columns=True)

path = "/PERLND/PQUAL/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "SDLFAC" not in df.columns:  # didn't read LAT-FACTOR table
        df["SDLFAC"] = 0.0
        df["SLIFAC"] = 0.0
        df["ILIFAC"] = 0.0
        df["ALIFAC"] = 0.0
        df.to_hdf(store, key=path, data_columns=True)

path = "/RCHRES/GENERAL/INFO"
if path in keys:
    dfinfo = pd.read_hdf(store, path)
    path = "/RCHRES/HYDR/PARAMETERS"
    if path in keys:
        df = pd.read_hdf(store, path)
        df["NEXITS"] = dfinfo["NEXITS"]
        df["LKFG"] = dfinfo["LKFG"]
        if "IREXIT" not in df.columns:  # didn't read HYDR-IRRIG table
            df["IREXIT"] = 0
            df["IRMINV"] = 0.0
        df["FTBUCI"] = df["FTBUCI"].map(lambda x: f"FT{int(x):03d}")
        df.to_hdf(store, key=path, data_columns=True)
    del dfinfo["NEXITS"]
    del dfinfo["LKFG"]
    dfinfo.to_hdf(store, key="RCHRES/GENERAL/INFO", data_columns=True)

path = "/RCHRES/HTRCH/FLAGS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "BEDFLG" not in df.columns:  # didn't read HT-BED-FLAGS table
        df["BEDFLG"] = 0
        df["TGFLG"] = 2
        df["TSTOP"] = 55
        df.to_hdf(store, key=path, data_columns=True)

path = "/RCHRES/HTRCH/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "ELEV" not in df.columns:  # didn't read HEAT-PARM table
        df["ELEV"] = 0.0
        df["ELDAT"] = 0.0
        df["CFSAEX"] = 1.0
        df["KATRAD"] = 9.37
        df["KCOND"] = 6.12
        df["KEVAP"] = 2.24
        df.to_hdf(store, key=path, data_columns=True)

path = "/RCHRES/HTRCH/PARAMETERS"
if path in keys:
    df = pd.read_hdf(store, path)
    if "MUDDEP" not in df.columns:  # didn't read HT-BED-PARM table
        df["MUDDEP"] = 0.33
        df["TGRND"] = 59.0
        df["KMUD"] = 50.0
        df["KGRND"] = 1.4
        df.to_hdf(store, key=path, data_columns=True)

path = "/RCHRES/HTRCH/STATES"
if path in keys:
    df = pd.read_hdf(store, path)
    # if 'TW' not in df.columns:  # didn't read HEAT-INIT table
    #    df['TW']    = 60.0
    #    df['AIRTMP']= 60.0

# apply defaults:
# JUST FOR TESTING WE OVERWRITE hsp_paths to ONLY CONTAIN the SNOW PERLND
path = '/PERLND/SNOW/STATES'
hsp_paths = {path:hsp_paths[path]}

for path in hsp_paths:
    if path in keys:
        df = pd.read_hdf(store, path)
        dct_params = hsp_paths[path]
        
        new_columns = {}
        for par_name in dct_params:
            if par_name == "CFOREA":
                ichk = 0
            
            if par_name not in df.columns:  # missing value in HDF5 path
                def_val = dct_params[par_name]
                if def_val != "None":
                    # df[par_name] = def_val
                    new_columns[par_name] = def_val
        
        new_columns = pd.DataFrame(new_columns, index=df.index)
        df1 = pd.concat([df, new_columns], axis=1)
        
        df1.to_hdf(store, key=path, data_columns=True)
    else:
        if path[-6:] == "STATES":
            # need to add states if it doesn't already exist to save initial state variables
            # such as the case where entire IWAT-STATE1 table is being defaulted
            if not "df" in locals():
                x = 1  # sometimes when debugging keys gets creamed, seems like an IDE bug
            for column in df.columns:  # clear out existing data frame columns
                df = df.drop([column], axis=1)
            dct_params = hsp_paths[path]
            for par_name in dct_params:
                def_val = dct_params[par_name]
                if def_val != "None":
                    df[par_name] = def_val
            df.to_hdf(store, key=path, data_columns=True)

# now see what the hdf has in it
pd.read_hdf(store, path)
store.close()

```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

make `update_uci` function to omit WDM updates #119

Case Study

Load the Libs and hdf5

Explore the dictionary

The test

Debugging Code

The Entire Code as command line runnable test

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

make update_uci function to omit WDM updates #119

Description

Case Study

Load the Libs and hdf5

Explore the dictionary

The test

Debugging Code

The Entire Code as command line runnable test

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

make `update_uci` function to omit WDM updates #119