Unable to save Table data to csv

[icarmel]

Hello Jeff,
My name is Idit and I'm Miryam Levy's daughter (from BIU).
In the BIU lab, they are using neurotic version 1.5.0 and unable to save the data displayed to a CSV file. It generated only the headers.
I debugged it a bit and it seems that the NeuroticWritableEpochSource is missing a source.
I'd like to propose a fix but have no permission to push a Pull Request.
Please find the patch diff. Thanks, Idit

commit 1335f505b6f9e3117352ec3db4ec476fdcbeea22
Author: Idit Levy Carmel <ilevycarmel@microsoft.com>
Date:   Wed Oct 4 09:50:28 2023 +0300

    change DataFrame table source in NeuroticWritableEpochSource

diff --git a/neurotic/gui/config.py b/neurotic/gui/config.py
index cc916cd..3e7377c 100644
--- a/neurotic/gui/config.py
+++ b/neurotic/gui/config.py
@@ -675,6 +675,20 @@ class EphyviewerConfigurator():
             epoch_view.params['xratio'] = self.metadata.get('past_fraction', 0.3)
             epoch_view.params['label_size'] = ui_scales[ui_scale]['channel_label_size']
 
+        ########################################################################
+        # DATAFRAME
+
+        self.annotations_dataframe = _neo_epoch_to_dataframe(seg.epochs, exclude_epoch_encoder_epochs=True)
+        if self.is_shown('data_frame') and len(self.annotations_dataframe) > 0:
+
+            data_frame_view = ephyviewer.DataFrameView(source = self.annotations_dataframe, name = 'Table')
+            if 'Events' in win.viewers:
+                win.add_view(data_frame_view, tabify_with = 'Events')
+            elif 'Video' in win.viewers:
+                win.add_view(data_frame_view, split_with = 'Video')
+            else:
+                win.add_view(data_frame_view, location = 'bottom', orientation = 'horizontal')
+
         ########################################################################
         # EPOCH ENCODER
 
@@ -693,6 +707,7 @@ class EphyviewerConfigurator():
             writable_epoch_source = NeuroticWritableEpochSource(
                 filename = _abs_path(self.metadata, 'epoch_encoder_file'),
                 possible_labels = possible_labels,
+                source = self.annotations_dataframe
             )
 
             epoch_encoder = ephyviewer.EpochEncoder(source = writable_epoch_source, name = 'Epoch encoder')
@@ -769,19 +784,6 @@ class EphyviewerConfigurator():
             else:
                 win.add_view(event_list, location = 'bottom', orientation = 'horizontal')
 
-        ########################################################################
-        # DATAFRAME
-
-        annotations_dataframe = _neo_epoch_to_dataframe(seg.epochs, exclude_epoch_encoder_epochs=True)
-        if self.is_shown('data_frame') and len(annotations_dataframe) > 0:
-
-            data_frame_view = ephyviewer.DataFrameView(source = annotations_dataframe, name = 'Table')
-            if 'Events' in win.viewers:
-                win.add_view(data_frame_view, tabify_with = 'Events')
-            elif 'Video' in win.viewers:
-                win.add_view(data_frame_view, split_with = 'Video')
-            else:
-                win.add_view(data_frame_view, location = 'bottom', orientation = 'horizontal')
 
         ########################################################################
         # FINAL TOUCHES
diff --git a/neurotic/gui/epochencoder.py b/neurotic/gui/epochencoder.py
index daa5503..04ae113 100644
--- a/neurotic/gui/epochencoder.py
+++ b/neurotic/gui/epochencoder.py
@@ -22,12 +22,13 @@ class NeuroticWritableEpochSource(WritableEpochSource):
     custom CSV column formatting and automatic file backup.
     """
 
-    def __init__(self, filename, possible_labels, color_labels=None, channel_name='', backup=True):
+    def __init__(self, filename, possible_labels, source=None, color_labels=None, channel_name='', backup=True):
         """
         Initialize a new NeuroticWritableEpochSource.
         """
 
         self.filename = filename
+        self.source = source
         self.backup = backup
 
         WritableEpochSource.__init__(self, epoch=None, possible_labels=possible_labels, color_labels=color_labels, channel_name=channel_name)
@@ -76,8 +77,13 @@ class NeuroticWritableEpochSource(WritableEpochSource):
             shutil.copy2(self.filename, backup_filename)
 
         df = pd.DataFrame()
-        df['Start (s)'] = np.round(self.ep_times, 6)                   # round to nearest microsecond
-        df['End (s)'] = np.round(self.ep_times + self.ep_durations, 6) # round to nearest microsecond
-        df['Type'] = self.ep_labels
+        if self.source:
+            df['Start (s)'] = np.round(self.source['Start (s)'], 6)                   # round to nearest microsecond
+            df['End (s)'] = np.round(self.source['End (s)'] + self.source['Duration (s)'], 6) # round to nearest microsecond
+            df['Type'] = self.source['Type'] 
+        else: 
+            df['Start (s)'] = np.round(self.ep_times, 6)                   # round to nearest microsecond
+            df['End (s)'] = np.round(self.ep_times + self.ep_durations, 6) # round to nearest microsecond
+            df['Type'] = self.ep_labels
         df.sort_values(['Start (s)', 'End (s)', 'Type'], inplace=True)
         df.to_csv(self.filename, index=False)

[icarmel]

df_change.log

[jpgill86]

Hi Idit,

If you'd like to propose a change to neurotic, the preferred method is the following:

1. Create a fork of the project under your GitHub account. You can do that by clicking here. You will have full permission to work on your fork.
2. Create a branch in your fork that will be dedicated to the change you are making.
3. Commit changes to that branch.
4. Open a pull request from the branch on your fork to the main neurotic repository.

I understand this may seem very complicated if you have never done it before. The advantage is that GitHub will automatically run neurotic's test suite, and I can more easily check that the proposed changes work. This off-loads work from me. Presently, I do really have time that I can devote to this.

Without actually testing your code, but just reading it, it looks to me like this is not a good change in its present form. The writable_epoch_source is needed for the Epoch Encoder, which is a separate widget panel from the annotations table that I know Miryam wants to export data from. So, the change you make may allow the annotations table to be exported, but it will break the Epoch Encoder. I think a different approach would be needed here.

Have you seen the script I provided to Miryam for exporting the contents of the annotations table? I will copy its contents below. I know a script is not as convenient as a button click in the application, but it should be able to export the table's contents.

import os
from pathlib import Path
import pandas as pd
import neurotic


################################################################################
# Change these values for your needs
metadata_file = 'metadata.yml'
dataset_list = [
    'IN VIVO / JG12 / 2019-05-10 / 002',
]
################################################################################

for dataset in dataset_list:
    print('Dataset:', dataset)

    metadata = neurotic.MetadataSelector(file=metadata_file)
    metadata.select(dataset)

    output_file = Path(metadata['data_file']).stem + ' TABLE.csv'
    output_file = Path(metadata['data_dir']) / output_file

    if output_file.exists():
        print('ATTENTION: The output file already exists:')
        print('    ', output_file)
        overwrite = input('Do you want to overwrite it? [y/n] ')
        if overwrite != 'y':
            print('OK, aborting')
            exit()

    exclude_epoch_encoder = input('Do you want to exclude epoch encoder entries from the output? [y/n] ')

    print('Loading data and generating table...')
    blk = neurotic.load_dataset(metadata, lazy=False)
    seg = blk.segments[0]
    epochs = seg.epochs

    print('Saving table...')
    df_list = []
    for ep in epochs:
        if exclude_epoch_encoder == 'y':
            ep = ep[ep.labels != '(from epoch encoder file)'];
        df_list.append(pd.DataFrame({
            'Start (s)': ep.times.rescale('s').round(4),
            'End (s)': (ep.times + ep.durations).rescale('s').round(4),
            'Duration (s)': ep.durations.rescale('s').round(4),
            'Type': ep.name,
            'Label': ep.labels,
        }))
    df_all = pd.concat(df_list).sort_values(['Start (s)', 'End (s)', 'Type', 'Label'])
    df_all.to_csv(output_file, index=False)

    print('Table successfully written to:')
    print('    ', output_file)
    print()

[icarmel]

Thank you, Jeff!
I've heard excellent feedback from my mom and team in the lab, who hold you in high regard and greatly appreciate your work and efforts.
The fix I did was a workaround to pass the source in constructor as WritableEpochSource values (represented by self instance) were empty.
I'll follow up with her regarding the attached script, and she'll take it from there.