Skip to content

Conversation

@Christian-B
Copy link
Member

This PR was motivated by looking at the https://github.com/SpiNNakerManchester/microcircuit_model and noticing that 75% of the time the script took to run was getting and printing data. With a large part of that due to the Creation of SpikeTrains.

This gives the option to write the data in csv format rather than pickle format.

This also includes a number of fixes independent of the csv writing.

  • Fixed the Population.last_id
    By changing the last_id returned by view.addPopulation method

  • Fixed the Population.describe which was too verbose Fixes Population.describe is too verbose #1278
    Changed abstract_pynn_model.py by adding name property (used by Population describe) and a describe method (used by standard PyNN)
    abstract_pynn_neuron_model.py - name based on model name
    Changed Population describe to celltype describe rather than vertex describe, and to only give the first neurons parameters. (more like what PyNN attempts)
    Depends on fix some issues with the describe() method - see #768 and #769 NeuralEnsemble/PyNN#770 to work 100%

  • DataPopopulation. Single place to get_data via renamed NeoBufferDatabase.get_full_block

  • Fixed neo.block metadata
    dt is now the timestamp as expected
    simulator name is at time of recording not time of use

The SpikeTrain init is now called with
units=quantities.ms,
dtype=numpy.float64,
sampling_rate=1000 / sampling_interval_ms * quantities.Hz

instead of
units='ms',
sampling_interval=sampling_interval_ms,
This change alone gives a 20% speed up due to how SpikeTrains work and fixes #1281

Moved the creation of a neo.block to NeoBufferDatabase (from recorder)

Added csv file writing
To Population and Population view based in file exstension
Via recorded to NeoBufferDatabase

Split NeoBufferDatabase into
interaction with the database - still NeoBufferDatabase
Creation of Neo Objects -> moved to NeoCsv

NeoCsv includes three sets of methods

  • insert for adding data (from either db or csf ) into Neo
  • csv for writing csv file
  • read for reading back the csv to a Neo object

For users the use pattern is
pop_1.write_data("test.csv")
neo = NeoCsv().read_csv("test.csv")
Which gives also the same neo Object as neo = pop_1.get_data("all")
-difference is if annotations are used they are strings in the neo which comes from csv

Added test for csv
Added test that neo.segments[0].filter(name='v')[0].times works

Tested by:
http://apollo.cs.man.ac.uk:8080/blue/organizations/jenkins/Integration%20Tests/detail/csv/3/pipeline

Speed testing this:
get_neo took 1.7910 seconds (from sqlite -> neo)
write_csv took 1.1397 seconds (from sqlite -> csv file)
read csv took 2.3950 seconds (from csv file -> neo)
write_pickle took 2.6362 seconds (from sqlite3 -> pkl file)
read pickle took 3.0520 seconds (from pkl file -> neo)

Also size
data.sqlite3 6.7 mb
csv 20.4 mb
pkl 22.2 mb

@rowleya rowleya merged commit 9ebb66e into master Feb 3, 2023
@rowleya rowleya deleted the csv branch February 3, 2023 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SpikeTrain expects sampling_rate not sampling_interval Population.describe is too verbose

3 participants