BF Event post-processing

[rburghol]

@benhdye - use this to put work and updates.

Approach 1: Trim from the front until slope of computed AGWRC goes to zero

Analyze how trimming changes computed AGWR

• Working in ben_trimming branch ( see https://github.com/HARPgroup/HARParchive/blob/ben_trimming/HARP-2025-2026/Event_trimming_function.R )
• Using event trimming script.R currently
• working on integration with event_indetification.R

Next Steps... AGWR Parameter Sensitivity & Threshold Refinement (notes from 10/21 meeting):

• Test parameter sensitivities
  • Current n = 5; try n = 7 and n = 10 (and n = 14 for extended tests)
  • Current p-value = 0.1; test p = 0.05 and p = 0.2
• Assess effects on AGWR
  • Evaluate how changing these parameters affects AGWR values overall.
  • Examine whether changes are more pronounced during low-flow vs high-flow periods.
• Compute population statistics
  • For each gage, calculate percentiles (e.g., 25th, 50th, 75th) and mean AGWR values.
  • Compare results across different parameter settings to quantify change.
• Visual and comparative analysis
  • Plot resulting AGWR vs flow to visualize parameter effects.
  • Use population statistics plots or tables to summarize results for each gage.

[benhdye]

I successfully implemented a function that will calculate the significance according to the mann-kendall test, and remove days from the event until the test reads no significance. The events are stored in a dataframe with a logic columns for each day showcasing whether it was trimmed or included in the event, which filtering out the trimmed events will remove them.

Here are a few examples of plotted events from the assessment report where the days that are red dots are the trimmed days:

In general, the function most commonly only trims a few days off the front of the event, which is what we would expect as they are stormflow influenced, but in some rare cases, like CS even 166, it may trim almost half off a 40 days event:

[rburghol]

@benhdye awesome - I agree, it is very much meets expectations. I would like to see:

• the statistics before and after trimming that accompanies these.
• A plot that can compare the points calculated with the original method, superposed on a plot of the points calculated with this method.
• descriptive statistics of these two larger data sets.

[benhdye]

Here are the comparisons between the p-values, hydrographs, and AGWR and delta AGWR before and after trimming:

MJ Group 94:

This group was determined to have no significance from the initial p value, so it was not trimmed.

MJ Group 152:

Strasburg 194:

CS group 62:

CS group 166:

Overall, trimming the first few days until no significance is found with a mann-kendall test seems to do a good job at eliminating stormflow influence, and leaving the events that are not. For example MJ event 94 was not trimmed and does not appear to be stormflow influenced, where as S 194 is trimmed and looks much closer to baseflow after.

[benhdye]

Branch: ben_trimming
Script: Drought_Trimming_Identifier.R
I adapted event_identification.R to work with trimmed events as well and ran it for each gage to get a before and after trimming results, which can be found in the following .csv files:

Cootes Store:

https://github.com/HARPgroup/baseflow_storage/blob/ben_trimming/CS_original_analysis.csv
https://github.com/HARPgroup/baseflow_storage/blob/ben_trimming/CS_trimmed_analysis.csv

Mount Jackson:
https://github.com/HARPgroup/baseflow_storage/blob/ben_trimming/MJ_original_analysis.csv
https://github.com/HARPgroup/baseflow_storage/blob/ben_trimming/MJ_trimmed_analysis.csv

Strasburg:
https://github.com/HARPgroup/baseflow_storage/blob/ben_trimming/S_original_analysis.csv
https://github.com/HARPgroup/baseflow_storage/blob/ben_trimming/S_trimmed_analysis.csv

Trimmed dataframe contains the trimmed data, which was subjected to the MK test.

[rburghol]

This is great @benhdye -- tagging @willprokopik for visualization and analysis of these files.

[willprokopik]

@rburghol and @benhdye just downloaded these files a few minutes ago and will begin some scatterplot analysis.

[rburghol]

That's great @willprokopik -- thanks!

[benhdye]

I started exploring how different p values effect the the trimming of events, I had some issues where some events would not work with certain MK p value threshold values. In the past few weeks I though I had been using a p value of 0.1, but I actually had it set to 0.05 which is the standard. I then tested p values of 0.01 and 0.1. I started working on a script to test different p value and how they remove events, and these are the preliminary results:

I believe the broken groups coming from the beginning of the data set where derivatives cannot be calculated because it utilized the first point of the dataset. This us more evidence that this methodology of using the MK test does not completely trim events from the dataset.

[benhdye]

Will and I plan to meet up to discuss the differences out two methods for trimming and decide on a happy medium. We will also explore how changing the p value threshold changes how many events are fully eliminated from the dataset as we move into a methodology that eliminates periods that do not meet the criteria.

[rburghol]

Thanks @benhdye @willprokopik - I would like you all to add the refinement ideas that came out of our meeting (I don't see them here but if they are point them out) -- to give you a start I recall 4 ideas that came out of today -- this is just a start, there may be more and you may devise more:

• Flag but don't delete events that fail more stringent tests
• Include Will's trim from end approach
• Keep the 5 day minimum event length (flag those with < 5)
• flag events whose calculated
  AGWRC after trimming is >= 1.0
• Areas of interest for analysis: 2002 CS and MJ; July 13 - Aug 6 1998 MJ

[willprokopik]

New MK Trimming Function

MK Trimming
This new function identifies a stable portion of each event by trimming from both the start and end based on MK test (same test Ben's function was running, now it trims from both sides). It tests all possible sub-windows of the AGWR series, running an MK test of each to find the longest segment where the p-value exceeds a chosen significance level (alpha)... meaning no sig trend remains. The resulting subset represents the stable "core" of the event, and it retains the start and end indices, p-values, and flags for whether the stability threshold was met. It records the window length and flags short windows if they fall below our chosen 5 day cutoff (for Cootes Store, it appears that just one event (101) was flagged for this (4 days)).

Here are some important trimming parameters to note:

trim_event_mk <- function(event_df, alpha = 0.10, 
                                    min_len_mk = 3,   # absolute minimum to test MK
                                    flag_len   = 5) { # flag if final window < this
  n <- nrow(event_df)

This takes care of the front/end MK trim approach and flagging events with < 5 days