ADD: bad_by_PSD method

[Randomidous]

PR Description

Building on @nabilalibou's work (#145), this PR adds functionality for finding bad channels using Power Spectral Density (PSD). This method is not part of the original MATLAB PREP pipeline and only runs when matlab_strict=False.

Detection Criteria

A channel is flagged as "bad-by-PSD" if either:

1. Abnormally high band power: The channel has an outlier z-score (>3.0) in any of three frequency bands:
   • Low (1-15 Hz): delta, theta, alpha
   • Mid (15-30 Hz): beta
   • High (30-45 Hz): gamma
2. 1/f violation: The high-frequency band has more power than the low-frequency band, which violates the typical 1/f spectral profile of EEG and often indicates muscle artifact or poor electrode contact.

Implementation Details

• PSD is computed using Welch's method over 1-45 Hz (configurable via fmin/fmax)
• Default frequency range excludes 50/60 Hz line noise
• Uses MAD-based robust z-scoring (scaled by 1.4826 to convert to SD units)
• Only flags positive z-scores (excess power), as abnormally low power may reflect normal topographic variation

Changes from previous PR (#145)

1. Detects only high PSD (not low), since low power can be normal
2. Uses MAD-based robust statistics instead of standard z-scoring
3. Splits spectrum into three frequency bands instead of total power
4. Adds 1/f violation criterion to catch muscle artifacts
5. Includes bad_by_psd in the bad channels dictionary
6. Fixes KeyError in Reference.robust_reference() by adding bad_by_psd to the noisy channels tracking dict

• closes: Add bad_by_PSD() #145

Merge Checklist

• the PR has been reviewed and all comments are resolved
• all CI checks pass
• (if applicable): the PR description includes the phrase closes #<issue-number> to automatically close an issue
• (if applicable): the changes are documented in the changelog changelog.rst
• (if applicable): new contributors have added themselves to the authors list in the CITATION.cff file

[sappelhoff]

I think in this step we lose some info. Imagine a channel that has an abnormally high PSD in low freqs, but lower than normal in high freqs --> the total power will then look like that of a channel that is medium in high and low 🤔

[sappelhoff]

just as a rule of thumb, we could try dividing into 3 "band power" bins (and only within bins summing up to total log psd):

1. 0-15Hz
2. 15-30Hz
3. 30-45Hz

a channel would be bad if the bin in 30-45Hz has more power than that of 0-15Hz.

a channel could also be bad if it is abnormal from other channels in any one of these three bins

a channel could also be bad if any ratio of bands is abnormal (1/2, 1/3, 2/3)

[Randomidous]

Added.

[Randomidous]

Removed third criterion

[Randomidous]

After testing with some artifact-ladden and somewhat clean data, I figured that the threshs need to be adapted or some criteria (eyeing bad_by_ratio atm) removed.

See plots below:
Noisy data:

Less noisy data (note still 7 channels removed):

[Randomidous]

Removing bad_by_ratio criterion:

What it did: For each channel, it computed the ratio of power between frequency bands (low/mid, low/high, mid/high), then flagged
channels where any of these ratios was unusual compared to other channels.

The rationale: The idea was to catch channels with abnormal spectral shape even if absolute power was normal. For example:

• A channel with normal total power but unusually flat spectrum (similar power across bands)
• A channel with an unusual bump in one band relative to others

Why it's problematic:

1. There's natural variation in spectral shape across the scalp (frontal vs occipital channels have different alpha power, etc. Imagine doing workload analysis and using this method)
2. Three separate ratio tests with OR logic means 3x the chance of false positives (I should have known better)
3. If a channel passes bad_by_band (absolute power is normal in all bands), the spectral shape is probably fine too - ratio
   deviations at that point are likely just normal topographic variation

The remaining criteria are more robust:

• bad_by_band: Catches genuinely abnormal power (too high or too low)
• bad_by_1f_violation: Catches channels where high-freq > low-freq power, which violates the fundamental 1/f characteristic of EEG and indicates noise/artifact

[codecov]

Codecov Report

❌ Patch coverage is 98.07692% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 98.08%. Comparing base (34d4773) to head (1292494).

Files with missing lines	Patch %	Lines
pyprep/find_noisy_channels.py	98.07%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #179      +/-   ##
==========================================
- Coverage   98.09%   98.08%   -0.01%     
==========================================
  Files           7        7              
  Lines         734      785      +51     
==========================================
+ Hits          720      770      +50     
- Misses         14       15       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Randomidous]

Without the ratio criterion, we still see a lot of channels being removed for noisy data. Some of these are clearly bad but others I'd rather keep in the dataset. So I will dig a bit deeper.

[Randomidous]

Without the ratio criterion, we still see a lot of channels being removed for noisy data. Some of these are clearly bad but others I'd rather keep in the dataset. So I will dig a bit deeper.

I mean, if a cleaning method fails on noisy data, then what's the point 🤔

[Randomidous]

Considering this atm:

• only mark union of bad channels?
• raise z-score thresh (on which grounds?)
• only consider positive z-scores? (since lower than average is not too sus)

[Randomidous]

Will check only flagging channels with abnormally HIGH power (positive z-scores). Excess power is more reliably indicative of problems (muscle artifact, noise, bad contact causing interference). Low power is ambiguous.
Getting closer and closer to the HF noise criterion.

[Randomidous]

This seems to work splendidly!

Below are bad channels only identified by bad_by_PSD in grey:

These are all bads by PSD (as compared to unique for PSD):

Running it on the same clean data from earlier only returns a single bad channel by PSD:

[Randomidous]

The current implementation seems promising and will be tested on a complete dataset next week :)

[sappelhoff]

nice work here @Randomidous looking forward to seeing the test results next week.

[sappelhoff]

@nabilalibou please feel free to have a look at this PR that is in the same vein of what you worked on in 2024.

Please also supply us with your details so we can add you to CITATION.cff, to give you some credit for getting this started at some point.

Follow this example, please:

    - given-names: Roy Eric
      family-names: Wieske
      affiliation: 'Biopsychology and Neuroergonomics, Technische Universität Berlin, Berlin, Germany'
      orcid: 'https://orcid.org/0009-0006-2018-1074'