Sufficiently open science

It’s difficult to disagree with the idea that psychologists (and other social scientists) need to be more open about their methods and data. Recently, a growing number of researchers have started being a lot more open about their methods and data (see the growing number of public projects on the Open Science Framework for evidence of this). I am very happy with these developments (along with many others) because of course being more transparent in how we do our research goes a long way in increasing the reliability of our empirical findings. I will not focus on these benefits here (but see this blog post for a great review of current open science initiatives and the various benefits of open science practices).

Rather, in this post I will  argue that though the open science movement is a wonderful development, a “sufficiently open science” (SOS) movement perhaps would be better. I want to consider the position that at this point in time — in this very special era we are currently living in — adopting SOS practices may actually be more beneficial overall for our field in the long run.

What do I mean by “sufficiently open science” practices? From my perspective, SOS practices means that:

  1. you provide minimum methods disclosure,
  2. you share minimum materials, and
  3. you share the minimum dataset underlying your claims.

That’s it! Absolutely no need to be more open than this, though of course that is completely fine if you so desire.

By “minimum methods disclosure”, I mean that you disclose the four categories of methods disclosure covered by Simmons et al.’s 21-word disclosure (and now required at Psychological Science and other journals). [UPDATE: Minimum methods disclosure actually involves a modified wording of the 21-word disclosure whereby an author needs to disclose only the measures that were analyzed with respect to the target research question, rather than all measures assessed. Thanks to Lorne Campbell for pointing this out.] Consistent with PLoS ONE’s new Data Sharing policy, by “minimal dataset”, I mean sharing “the dataset used to reach the conclusions drawn in the manuscript with related metadata and methods, and any additional data required to reproduce the reported study results in their entirety.” Finally, “minimum materials” means sharing the essential materials and procedures needed for a competent independent researcher to execute a diagnostic direct replication. (And if you’re willing to share the minimum dataset and materials with one interested researcher, then why not take 5 minutes and post these publicly on Figshare.com or the OSF so that if someone else emails you about those things, your job is already done!)

I contend that it’s more profitable — at this point in time — to strive for SOS practices because it is more feasible and pragmatic for time-starved researchers, all of whom are caught in a chaotic and rapidly-changing research landscape. Also, it’s a great stepping stone toward broader openness, which can happen in a more natural and gradual manner. And because SOS practices are more feasible, it means that reaching a tipping point where the majority of researchers are being more open may happen much sooner.

Take-home message: Practice “sufficiently open science” and spread the word to any colleagues you know who may still be on the fence about the open science movement!

Oh and here’s a visual depiction of what I’m trying to say, comparing features of the status quo, sufficiently open science (SOS), and open science (OS) approaches.

approaches-compared

*According to current methods disclosure rates at PsychDisclosure.org (see also LeBel et al., 2013).

**According to Wicherts, Borsboom, Kats, & Molenaar (2006).

Advertisements

2 comments

  1. Regarding SOS, it seems as though all materials that were tested in order to produce the results in the ms need to be disclosed–not simply the materials that ended up in the final results. This may be implied of course in your approach. It would be ideal, then, for people to register/post all materials to be used in a given data analytic plan in advance of testing hypotheses, and NOT to post materials post-analysis. Is that what you are suggesting?

  2. You are correct and this is very important. I have now updated the post to clarify this ambiguity, adding the following text, thanks!
    [UPDATE: Minimum methods disclosure actually involves a modified wording of the 21-word disclosure whereby an author only needs to disclose all measures that were analyzed with respect to the target research question, rather than all measures assessed. Thanks to Lorne Campbell for pointing this out.]

    And I like your thinking regarding the implications of the SOS approach for pre-registration. That’s precisely what I’d suggest for researchers looking to be more “pragmatic” in this context. That is, publicly pre-register all measures and materials to be used in a pre-specified data analytic plan, but then just publicly post (on a public repository) the measures and materials that actually ended up being analyzed with respect to the target research question.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s