Past, Present and Future of Open Science (Emergent session): Pre-registration, Registered reports

[jsheunis]

Pre-registration, Registered reports

By Rémi Gau, Université catholique de Louvain, Louvain-La-Neuve, Belgique

• Theme: Past, Present and Future of Open Science
• Format: Emergent session

Abstract

Planned time

Wednesday 1rst of July ; 11AM (UK time)

Participants:

1. Rémi gau
2. Rotem Botvinik-Nezer
3. Chris Chambers
4. Gustav Nilsonne
5. Dustin Moraczewski
6. Johannes Algermissen
7. Olivia Guest
8. TBD

We want to create a space to have a discussion about some of the following points (and others related):

• Introduce pre-registration and registered reports and explain the difference between them.
• Pre-registration in practice in MRI or M/EEG: why has pre-registration not (yet) taken off for neuroimaging? Special difficulties, hurdles, ...
• What can we do to facilitate pre-registration? Thinking especially about training opportunities and ECR perspectives.
• Does pre-registration help with “the file drawer effect”? Talk about some of the studies that have started looking into the effect of registered reports when it comes to publish negative results.
• Are pre-regs for everyone? There has been some criticism of the “pre-reg hype” (in part coming from the cognitive modelling community) and we think it would be nice to have a live discussion about this.
• Confirmatory vs. exploratory research: not all research might be ready for power analyses / confirmatory testing (cf exploratory reports at Cortex)
• What should be included in a pre-registration and what should not? Consider readability, perusability by reviewers.
• What are the most important things to prereg in neuroimaging (might be different also between MRI and EEG)? Where is it most important to constrain researcher degrees of freedom?
• What are the incentives to pre-register?
  "open washing" : saying that something is pre-reg:ed should not automatically mean it is higher quality or more true.

Useful Links

https://docs.google.com/document/d/1FMItbzwfVEz_pgrpdIg9dn1uPOriuaautLVbK9V0uGw/edit?usp=sharing

Tagging @Remi-Gau

[johalgermissen]

Some useful links to get started on pre-registration:

Pre-registration

OSF

• Overview pre-registration templates on the Open Science Framework (OSF): https://osf.io/zab38/wiki/home/
  fMRI pre-registration template by Jessica Flannery: https://osf.io/dvb2e/

As-predicted.org

• As-predicted webpage: https://aspredicted.org/
• As-predicted motivation: [64] How To Properly Preregister A Study: http://datacolada.org/64

Further templates (in progress)

• EEG pre-registration template started at SIPS 2019 (version: "MPI-CBS"), in progress, contact @mariellapaul and @aschetti: https://docs.google.com/document/d/1RmlHcWJcWMErZzjJgz9Q0rqsy-E9Vs9DYeKdvX9cawU/edit
• EEG pre-registration spin-off of the one started at SIPS 2019 (version: "Donders"), in progress, contact @johalgermissen: https://docs.google.com/document/d/160DcKWpeYmoc-aEYZaoKN43534wvTyMJEFBQdahGBjU/edit?usp=sharing
• fMRI -registration template (version. "MPI-CBS"), in progress, contact @mariellapaul: https://docs.google.com/document/d/1ohwRNKNO7X2gIgS8qVmlmc0C8XvHvtsXKcP4LWjkrQI/edit#heading=h.u5fvu02msef5
• Shiny-App to automatically write MEG/ EEG pre-registration, started at OHBM Hackathon 2020, in progress, contact @johalgermissen: https://johalgermissen.shinyapps.io/mapMEEG_proto/

Overview existing pre-registrations in neuroimaging (in progress)

• Zotero library containing published pre-registered studies: https://www.zotero.org/groups/479248/osf/collections/VKXUAZM7
• List of existing fMRI/ MEG/ EEG pre-registrations (started at SIPS 2019, in progress): https://docs.google.com/spreadsheets/d/1_R3UBKvT4vEDOAPXxR8W98QGvZMhpuyxlvYzIGekf0w/edit?usp=drive_web&ouid=107034845673249190997
• List of existing MEG/ EEG pre-registrations (started at OHBM Hackathon 2020, in progress): https://docs.google.com/spreadsheets/d/1-tTgpDOKOAUG-zdfqAnbBfPOsCDvGC5wzaOLKdBAcWc/edit?usp=sharing
  • Further pre-registrations that still need to be coded: https://docs.google.com/document/d/1FT6bg89dUQscQ1JNm3mFaWht5SkfIhQeaBkGlbT6EO4/edit?usp=sharing

Further references on pre-registration:

• Crüwell, S., Stefan, A. M., & Evans, N. J. (2019). Robust standards in cognitive science. Computational Brain & Behavior, 2(3-4), 255-265. https://link.springer.com/article/10.1007/s42113-019-00049-8
• Devezer, B., Navarro, D. J., Vandekerckhove, J., & Buzbas, E. O. (2020). The case for formal methodology in scientific reform. bioRxiv. https://www.biorxiv.org/content/10.1101/2020.04.26.048306v1
• Lee, M. D., Criss, A. H., Devezer, B., Donkin, C., Etz, A., Leite, F. P., ... & Vandekerckhove, J. (2019). Robust modeling in cognitive science. Computational Brain & Behavior, 2(3-4), 141-153. https://link.springer.com/article/10.1007/s42113-019-00029-y
• Navarro, D.J. (2019). Between the devil and the deep blue sea: tensions between scientific judgement and statistical model selection. Computational Brain & Behavior, 2(1), 28–34. https://link.springer.com/article/10.1007/s42113-018-0019-z
• Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., ... & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23(10), 815-818. https://www.sciencedirect.com/science/article/pii/S1364661319301846
• Szollosi, A., Kellen, D., Navarro, D. J., Shiffrin, R., van Rooij, I., Van Zandt, T., & Donkin, C. (2019). Is preregistration worthwhile. Trends in Cognitive Sciences, 24(2), 94-95. https://www.sciencedirect.com/science/article/pii/S1364661319302852

Registered reports

Evolution of registered reports

• Chambers, C. D. (2013). Registered reports: a new publishing initiative at Cortex. Cortex, 49(3), 609-610. https://www.sciencedirect.com/science/article/pii/S0010945212003735
• Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered reports: realigning incentives in scientific publishing. Cortex, 66, A1-A2. https://www.sciencedirect.com/science/article/pii/S0010945215001136
• Chambers, C. D., Forstmann, B., & Pruszynski, J. A. (2017). Registered reports at the European Journal of Neuroscience: consolidating and extending peer-reviewed study pre-registration. European Journal of Neuroscience, 45(5), 627-628. https://onlinelibrary.wiley.com/doi/full/10.1111/ejn.13519
• Hardwicke, T. E., & Ioannidis, J. P. (2018). Mapping the universe of registered reports. Nature Human Behaviour, 2(11), 793-796. https://www.nature.com/articles/s41562-018-0444-y
• Chambers, C. D., & Mellor, D. T. (2018). Protocol transparency is vital for registered reports. Nature Human Behaviour, 2(11), 791-792.https://www.nature.com/articles/s41562-018-0449-6
• Chambers, C. (2019). What’s next for registered reports?. Nature. https://www.nature.com/articles/d41586-019-02674-6

Overview existing registered reports

• Comparison of journals offering Registered Reports: https://docs.google.com/spreadsheets/d/1D4_k-8C_UENTRtbPzXfhjEyu3BfLxdOsn9j-otrO870/edit#gid=0
• OSF search function for Stage 1 registered reports: https://osf.io/registries/discover?provider=OSF&type=Registered%20Report%20Protocol%20Preregistration
• Zotero library containing registered reports: https://www.zotero.org/groups/479248/osf/collections/KEJP68G9

Evaluation of registered reports

• Allen, C., & Mehler, D. M. (2019). Open science challenges, benefits and tips in early career and beyond. PLoS biology, 17(5), e3000246. https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000246
• Scheel, A. M., Schijen, M., & Lakens, D. (2020). An excess of positive results: Comparing the standard Psychology literature with Registered Reports. psyArXiv. https://psyarxiv.com/p6e9c

[Remi-Gau]

Holy monkey you were not kidding about having that ready to go.

Will mention this recent preprint:
The case for formal methodology in scientific reform, Berna Devezer, Danielle J. Navarro, Joachim Vandekerckhove, Erkan Ozge Buzbas. https://www.biorxiv.org/content/10.1101/2020.04.26.048306v1.full.pdf+html

[complexbrains]

This event has been scheduled to be run on 31.06.2020, 10:00- 11:00 UTC

For more information, please go to https://ohbm.github.io/osr2020/schedule/emea

[Remi-Gau]

Sorry for the cross-posting.

A bit late but here would be a basic structure for this session.

I volunteer to moderate the discussion (I have opinions about the topic but I would rather hear what y'all have to say).

• I introduce the session briefly (to give a tiny bit of context)
• everyone introduces themselves briefly (who, where)
• have Chris give an "elevator-pitch" definition on pre-reg VS reg-reports so everyone is clear on definitions
• open the floor to the discussion and we try to talk some of the following (shorter version of the github "abstract" bullet points):
  • Pre-registration in practice in MRI or M/EEG: why has pre-registration not (yet?) taken off for neuroimaging ?
  • What are the most important things to prereg in neuroimaging ? The need for pre-registration templates ?
  • What can we do to facilitate pre-registration ? Incentives to pre-register ?
  • Does pre-registration improve things? with “the file drawer effect”? with publishing negative results? Other benefits of prereg (e.g pedagogical)?
  • Pre-registration and "open washing" : pre-reg as virtual signaling ?
  • What studies are more suited to pre-regs ? Confirmatory vs. exploratory ? Modelling work ?

In terms of tone let's try to go for a discussion where we try to understand each other's view point rather than "being right".

In terms of coverage let's try to strike a good balance between not going too far down a single rabbit hole but to cover several of those points.

[InquisitiveVi]

Will it be possible to post all the links shared during the discussion over here?

[Remi-Gau]

@InquisitiveVi
Yes will try to keep track and add them

[johalgermissen]

Links that came up during the discussion online and in the subsequent discussion on Mattermost:

Pre-registration

• McPhetres, J. (2020, June 1). What should a preregistration contain?. https://doi.org/10.31234/osf.io/cj5mh

Influence of pipeline decisions on conclusions

• Nørgaard, M., Ganz, M., Svarer, C., Frokjaer, V. G., Greve, D. N., Strother, S. C., & Knudsen, G. M. (2019). Different preprocessing strategies lead to different conclusions: A [11C] DASB-PET reproducibility study. Journal of Cerebral Blood Flow & Metabolism, https://doi.org/10.1177/0271678X19880450.
• Nørgaard, M., Ozenne, B., Svarer, C., Frokjaer, V. G., Schain, M., Strother, S. C., & Ganz, M. (2019, October). Preprocessing, Prediction and Significance: Framework and Application to Brain Imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 196-204). Springer, Cham. https://link.springer.com/chapter/10.1007/978-3-030-32251-9_22

Open Science badges

• Criteria for the open science badges: https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/

Theory in cognitive science

• Guest, O., & Martin, A. E. (2020, February 19). How computational modeling can force theory building in psychological science. https://doi.org/10.31234/osf.io/rybh9
  • Respective Twitter thread: https://twitter.com/o_guest/status/1230144174751698946
• Cooper, R. P. & Guest, O. (2014). Implementations are not specifications: specification, replication and experimentation in computational cognitive modeling. Cognitive Systems Research. https://doi.org/10.1016/j.cogsys.2013.05.001
• Guest, O. & Rougier, N. P. (2016). Dialogue: What is Computational Reproducibility?. IEEE CIS Newsletter on Cognitive and Developmental Systems. https://openlab-flowers.inria.fr/uploads/default/original/1X/65addc14bb2a6a7feaf7690865fa3708d5b0990f.pdf
• Miłkowski, M., Hensel, W. M., & Hohol, M. (2018). Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. Journal of Computational Neuroscience, 45(3), 163-172.
• Guest, O., Caso, A. & Cooper, R.P. (2020). On Simulating Neural Damage in Connectionist Networks. Computational Brain & Behavior.(2020). https://doi.org/10.1007/s42113-020-00081-z
  • Quote: "However, making code available is not sufficient, in and of itself, to avoid a replication crisis within cognitive modelling. Most obviously, in addition to raw code one also needs values of all training parameter and the complete set of training items. But even this does not guarantee an understanding of what specific elements of a model (or indeed a training set) are critical to capturing the effects of interest. Equally, it does not prevent behaviour of an implementation being attributed to features that are in fact implementation details and not causally relevant to the generation or production of that behaviour."

Model-based cognitive neuroscience

• O'Doherty, J. P., Hampton, A., & Kim, H. (2007). Model‐based fMRI and its application to reward learning and decision making. 'Annals of the New York Academy of Sciences_, 1104(1), 35-53.
  https://doi.org/10.1196/annals.1390.022
• Palmeri, T. J., Love, B. C., & Turner, B. M. (2016). Model-based cognitive neuroscience. Journal of Mathematical Psychology. https://doi.org/10.1016/j.jmp.2016.10.010
• Guest, O. & Love, B. C. (2017). What the Success of Brain Imaging Implies about the Neural Code. eLife. https://doi.org/10.7554/eLife.21397
• Love, B. C. (2019). Model comparison, not model falsification. Behavioral and Brain Sciences. (Commentary on Rahnev & Denison). https://doi.org/10.1017/S0140525X18001516
• Mok, R. & Love, B. C. (2019). A non-spatial account of place and grid cells based on clustering models of concept learning. Nature Communications. https://doi.org/10.1038/s41467-019-13760-8
• Love, B. C. (2020). Model-based fMRI analysis of memory. Current Opinion in Behavioral Sciences, 32, 88-93. https://doi.org/10.1016/j.cobeha.2020.02.012