-
Notifications
You must be signed in to change notification settings - Fork 47
Ability to csv file and some Neo interaction changes #1283
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was motivated by looking at the https://github.com/SpiNNakerManchester/microcircuit_model and noticing that 75% of the time the script took to run was getting and printing data. With a large part of that due to the Creation of SpikeTrains.
This gives the option to write the data in csv format rather than pickle format.
This also includes a number of fixes independent of the csv writing.
Fixed the Population.last_id
By changing the last_id returned by view.addPopulation method
Fixed the Population.describe which was too verbose Fixes Population.describe is too verbose #1278
Changed abstract_pynn_model.py by adding name property (used by Population describe) and a describe method (used by standard PyNN)
abstract_pynn_neuron_model.py - name based on model name
Changed Population describe to celltype describe rather than vertex describe, and to only give the first neurons parameters. (more like what PyNN attempts)
Depends on fix some issues with the
describe()method - see #768 and #769 NeuralEnsemble/PyNN#770 to work 100%DataPopopulation. Single place to get_data via renamed NeoBufferDatabase.get_full_block
Fixed neo.block metadata
dt is now the timestamp as expected
simulator name is at time of recording not time of use
The SpikeTrain init is now called with
units=quantities.ms,
dtype=numpy.float64,
sampling_rate=1000 / sampling_interval_ms * quantities.Hz
instead of
units='ms',
sampling_interval=sampling_interval_ms,
This change alone gives a 20% speed up due to how SpikeTrains work and fixes #1281
Moved the creation of a neo.block to NeoBufferDatabase (from recorder)
Added csv file writing
To Population and Population view based in file exstension
Via recorded to NeoBufferDatabase
Split NeoBufferDatabase into
interaction with the database - still NeoBufferDatabase
Creation of Neo Objects -> moved to NeoCsv
NeoCsv includes three sets of methods
For users the use pattern is
pop_1.write_data("test.csv")
neo = NeoCsv().read_csv("test.csv")
Which gives also the same neo Object as neo = pop_1.get_data("all")
-difference is if annotations are used they are strings in the neo which comes from csv
Added test for csv
Added test that neo.segments[0].filter(name='v')[0].times works
Tested by:
http://apollo.cs.man.ac.uk:8080/blue/organizations/jenkins/Integration%20Tests/detail/csv/3/pipeline
Speed testing this:
get_neo took 1.7910 seconds (from sqlite -> neo)
write_csv took 1.1397 seconds (from sqlite -> csv file)
read csv took 2.3950 seconds (from csv file -> neo)
write_pickle took 2.6362 seconds (from sqlite3 -> pkl file)
read pickle took 3.0520 seconds (from pkl file -> neo)
Also size
data.sqlite3 6.7 mb
csv 20.4 mb
pkl 22.2 mb