Do we stratify continuos target variables?

[kjappelbaum]

There is a lot of debate around this, see, e.g. the threads in sklearn.

Some thoughts:

• this is already a weak source of data leakage as we need to consider the full dataset for binning. This is also mentioned in this stackexchange thread:

Stratified Cross-validation violates the principal that the test labels should never have been looked at before the statistics are calculated, but this is generally thought to be innocuous as the only effect is to balance the folds. Arguably the main reason stratification is important is to address defects in the model training process, as the model could be too easily biased by over- or under-representation of classes.

• Some R-packages (including caret, or in splittools) do it by default
• There is some debate (see sklearn discussions) about the best way of implementing: binning (e.g. KBinsDiscretizer) or simply sorting and assigning every nth sample to a specific set. On top of that, there's also an interleaved or Venetian Blinds cross-validation approach that is implemented by the pls package. Diago et al (2018)
• A comment from Max Kuhn in the sklearn thread:

The text that @amueller was probably referring to was here:
> When the outcome is numeric, artificial strata can be constructed based on the quartiles of the data. For example, in the Ames housing price data, the quartiles of the outcome distribution would break the data into four artificial groups containing roughly 230 houses. The training/test split would then be conducted within these four groups and the four different training set portions are pooled together (and the same for the test set).
The idea is to make the resulting distributions of the target as close as possible (esp in the tails).
> Indeed I am not sure about the value of our default choice either.
Unless your data set size is really small (say 2-10 samples per bin) I have never seen any harm in this (it is the default in caret). For larger data sets, the risk of getting discordant distributions is pretty low.
> Also I think we could do a better on how to use the various options of our CV tools to detect breakage in the i.i.d. assumption.
I agree that with data sets with a strong time-effect, this could go badly (but any resampling method that doesn't account for seasonal/temporal patterns would be some level of problematic).

• There might be some more insight in this paper. However, glancing through it, itself does not compare to unstratified splits.