Malet release v0.2.1

🚀 Changes

• Experiment will now skip any erroneous config and continue to the next config.
• plot.draw_metric now return additional dataframe (df, *prior_returns) used for plotting.
• Command line argument -colors in malet-plot now takes in a name or list of names of a matplotlib colormap instead of '' or 'cont'.

✨ Features

Experiment tool updates

• Add malet.utils.QueuedFileLock and add to ExperimentLog to improve stability of queueing for grid parallelism. Add attribute use_filelock=False for option to enable filelocking whenever the tsv file is accessed.
• Add timeout option as keyword argument (timeout=None) in Experiment.
• Add error message when reading tsv fails in ExperimentLog.parse_tsv
• Add dropna kwarg to ExperimentLog.melt_and_explode_metric for dropping NaN valued rows.

Plot updates

• Add scatter and scatter_heat mode. Here, no processing except filtering will occur.
• Save tsv of metrics used for plotting. (Will have the same file name as the plot)
• Add command line arguments to
  • set fields for multiple column/row-wise subplots (-multi_plot_fields),
  • set field to animate over (-animation_field),
  • repeat, skip, shift the list of colors (-colors_rep_skip_shift),
  • set order of string fields (-field order),
  • set label and scale for colorbar (-zlabel, -zscale), and
  • set marker size (-marker_size).
• Add 'last' and 'best' options for 'step' field in -filter and -best_ref_x_fields when 'step' is in x_field.
• Add different styles other than colors (marker, linestyle) for different multi-line fields with improved legend.
• Add color bar for heatmap and scatter_heat mode.
• Various improvements to plot processing status and summary message.
• Improve error message for processed log filled with NaN to also print the actual dataframe.

🩹 Bug Fixes

• Raise error when grid fields have different number of groups in ConfigIter
• Fix ConfigIter and ExperimentLog when there are no grid in exp_config.yaml. This also fixes malet-plot.
• Fix xscale, yscale to be setted using plot_config.yaml in malet-plot.
• Correctly filter None values in plot_utils.data_processor.select_df
• Fix filtering 'total_step' conflicting with auto-filtering 'step' in plot_utils.data_processor.avgbest_df.
• Fix *exclude_fields argument in `plot_utils.data_processor.homogenize_df.
• Fix annotating nan-valued grid for plot_utils.plot_drawer.ax_draw_heatmap.
• Fix processing nan value in utils.str2value.

Malet release v0.1.13

Changes

• Cannot pass int split as string total_splits and curr_splits in Experiment.

Features

• Add option to filter out certain values alongside with filtering in at malet-plot (add '!' at the end of the field as '-filters field! v1 v2 ...').

Bug Fixes

• Add error message for overlapping fields in static and grid configs in ConfigIter. (some functionality in ExperimentLog breaks without this assertion.)
• Fix error when total_splits and curr_splits are passed in as int in Experiment.

Malet release v0.1.12

Bug Fixes

• Fix experiment skipping failed cases and executing already running cases.

Malet release v0.1.11

🚨 Fatal bug (removed from pypi)

• Experiment skipping failed cases and executing already running cases, which should be (and has been) the other way around.

Changes

• Change method name get_log_checkpoint into get_metric_info in Experiment, which now returns empty dictionaries if the log of matching config is not in the experiment log.

Features

• Add method to set all status R' in the tsv file’ into 'F' in Experiment.
• Add option to set status in Experiment.update_log.
• Add RunInfo class for various time tracking for Experiment.
• Add x, y scale option in malet-plot.
• Annotate standard deviation as mean±std in malet-plot.

Bug Fixes

• Fix run duration tracking not updated when calling update_log.
• Fix NaN checking in Experiment.get_metric_info (formerly Experiment.get_log_checkpoint).

Malet release v0.1.10

Features

• Add error message for non-matching static_config between ConfigIter and ExperimentLog in Experiment.

Bug Fixes

• Fix duration aggregation in Experiment.run
• Fix NaN checking in Experiement.get_log_checkpoint

Malet release v0.1.9

Changes

• Change arguments in Experiment.update_log from (metric_dict, config) to (config, **metric_dict).
• Change arguments and return value in Experiment.get_log_checkpoint from (config, empty_metric) → metric_dict to (config) → (metric_dict, info_dict).

Features

• Add train duration information and git info in experiment.Experiment.

Bug Fixes

• Fix Experiment not recording datetime and status.
• Fix Experiment not to delete metrics when checkpointing.
• Reflect ExperimentLog.add_result argument change to Experiment.update_log.

Malet release v0.1.8

Changes

• Remove info_field attribute in ExperimentLog due to redundancy. Please migrate all info_field to metric_field.
• Change method name get_metric_and_info → get_metric in ExperimentLog
• Remove kwarg: infos and change type of kwarg: metric from dict to unpacked keyword arguments.

Bug Fixes

• Fix str2value to process inf and return string if literal_eval throws an exception.
• Fix ExperimentLog.add_result for cases when there are two grid fields. (Dataframe.loc and multiindex edge case.)
• Fix isin to only check matching config in static configs.

Malet release v0.1.7

Changes

• Change method name explode_and_melt_metric → melt_and_explode_metric and its kwarg epoch → step in ExperimentLog.
• Compute function (arg: fn) is no longer mapped onto list fields when using ExperimentLog.add_conputed_metric.
• Change module structure of malet.plot_utils.

Features

• Can now parse metric of any nested-structure of standard python types (strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None and Ellipsis) from tsv file in ExperimentLog using ast.literal_eval.
• Add mode (curve_best) for plotting star marker for best performing among x.
• Add annotations to heatmap in malet-plot.
• Filter epoch by range in malet-plot.
• malet-plot can process list fields with various lengths.
• Automatic markevery setting depending on number of x-values in curve modes in malet-plot.
• Add arguments for title, x-label, y-label, font size in malet-plot.
• Filter log.df prior to melt_and_expand_metric for faster processing in plot.draw_metric.
• Add kwarg save to ExperimentLog.merge_folder.
• Better error messages
  • ExperimentLog: error message for overlapping field names when instantiating
  • plot_utils.data_processor.select_df: error messages for non-existent field, field_value, and empty dataframe before/after filter.

Bug Fixes

• Plotting when x-field is 'step' (previously 'epoch')
  • Fix from plotting only last step to all steps (wrong argument in explode_and_melt_metric call).
  • Fix error when 'best_ref_x_fields' is not specified.
• Annotate only above the marks in curve mode in malet-plot.
• Apply str2value to indices (grid fields) when loading tsv in ExperimentLog.from_tsv.
• Fix ExperimentLog.add_derived_index to handle tuples.

Malet release v0.1.5

Changes

• Experiment.resplit_logs now automatically finds log and config path from experiment folder.
• malet-plot accepts space-seperated lists in multi_line_fields, x_fields, and best_ref_x_fields.
• Change in ExperimentLog.merge from using pd.merge to `pd.concat’.

Features

• Better command line messages for malet-plot with rich library.
• Add bar plot and heatmap in malet-plot
• Cast list configs into tuple when adding configs to dataframe in ExperimentLog for hashablilty.
• Add option parse_str in ExperimentLog.parse_tsv to parse string in the pd.DataFrame entry into lists and tuples or keep it as string.
• Add method (ExperimentLog.add_computed₎_metric, ExperimentLog.add_derived_index) to compute other metric and indices derived from existing metric and indices.
• Add remove_metric and remove_index in `ExperimentLog’.
• Remove str, NaN and change metric_value column to float in ExperimentLog.explode_and_melt_metric. This allows more flexable config gridding.
• Add resplitting for ExperimentLogs.
• Add option to specify annotate field, figure size, and plot style in flags for malet-plot .
• Add save_path kwarg in merge methods in ExperimentLog.

Bug Fixes

• Fix ExperimentLog.merge_tsv by not parsing strings into list when reading tsv.
• Fix str2value to process scientific notations (e.g. 1.200004e+12) and nan
• Fix ExperimentLog.explode_and_melt_metric to stop raising exceptions when there are no list fields, and handle list fields with NaN entries
• Fix malet-plot throwing errors from having multiple metric fields when using best_ref_…_field.

Malet release v0.1.4

Features

• Automatic fail status logging using try-catch
• Merge add_configs_only method with add_result in ExperimentLog
• Improved shell messages when plotting (plot info, saved path)
• Better plot save names (add various plot configs to prevent overwrites)
• Add plot option to select optimal value direction between min or max

Bug Fixes

• Proper sorting when plotting over list-valued field
• Fix str2value integer and float error
• Fix ExperimentLog.from_tsv method call in Experiment