Raise research standards for *all* research, not just replication work

Great post by Siminie Vazire on whether independent replication work should be held to a higher standard than original research. I posted a comment to her post arguing that replication studies should definitely *not* be held to higher standards because sloppy original research has much more potential to cause damage overall when you consider the far reaching negative consequences original research can have in terms of wasted time, resources, and opportunity costs when large number of labs try to extend false positive findings.

In this post, I want to expand a bit more on these thoughts.

First, I want to make it clear that I agree that research standards for replication work should be very high, because replication work is very crucial to the scientific process! However, original research should also be held to that same very high standard because of the immeasurably high amount of damage unreliable original findings can have on a scientific literature. Highly provocative original findings will typically incite a large number of labs to try and extend those findings with the hope of learning deeper insights about those empirical phenomena. If the original findings are unreliable, however, this can lead to an enormous amount of wasted time and resources by the labs trying to extend those original findings. Furthermore, we shouldn’t forget about opportunity costs. To add insult to injury, unreliable original findings also impose immense opportunity costs for the (often) very smart researchers in those other labs that could have been working on *other* research questions that would have had a higher probability of yielding cumulative knowledge.

For example, the original research by Bargh et al. (1996) and Dijksterhuis et al. (1998) led to a gargantuan wave of research by hundreds of labs around the world attempting to shed light on the mechanisms of those social priming effects. Likely thousands of studies have been executed in this spirit (most of them likely file-drawered). Only time will tell, but it might turn out to be the case that all of the time, money, and resources invested in these efforts will not have yielded much fruit in return.

Take-home message: Yes let’s raise research standards for replication work because replication work is so crucial to ensuring the self-corrective process of science. But let’s remember that research standards for original work needs to be substantially raised as well. Hence, we definitely need to raise research standards, but we need to do this for *all* empirical research in psychology.

Visual and quantitative depiction of my take-home message:

Image

Advertisements

2 comments

  1. Great title for your blog!

    It reflects an attitude change that is necessary to resolve the crisis; from evidencing a scientific claim to trying everything possible to debunk a theory which will gain credibility if such endeavours fail.

    And yes… I think you are right to comment on the tendency among some groups of authors to demand and prescribe higher standards for replication researchers, while an “original research recipe” would be a more appropriate place to start.

  2. Thanks! But I think I should maybe put a brief explanation of my title somewhere on the blog because it may not make sense to most social scientists. It’s actually part of a Feynman quote which states: “We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress”!

    In other words, the quicker you prove yourself wrong, the quicker you must modify your theory and/or auxiliary assumptions, or methods, or meta-theoretic framework to better understand your target phenomenon. But if you don’t know you’re wrong, then of course there’s no need to modify anything and no progress is made.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s