replicability

High-powered direct replications of social psychology findings (for in press paper; out-of-date)

***IMPORTANT NOTE***: This list was compiled on October 13, 2015 solely for an in press paper at JPSP, to be referenced as additional replications of social psych findings **beyond** large-scale replication efforts such as RP:P, Social Psych special issue, ML1, and ML3 and was not meant to be disseminated widely. Hence, this list is completely out-of-date. For a more systematic effort to track replications in psychology, see Curate Science.

The table below lists successful (n=3) and unsuccessful (n=111) high-powered direct replications of social psychology findings (known to us on October 13, 2015). For simplicity, only replications with statistical power >= 80% to detect an effect size as large (or larger) than the original finding are included (citation counts according to Google Scholar, retrieved October 2015). This list was tabulated as additional evidence to support the broader position that the current incentive structure in social psychology is not conducive to generating cumulative knowledge in light of several meta-scientific investigations revealing low replicability rates of social psychology findings (e.g., Reproducibility Project: 76% replication failure rate of social psychology studies; Social Psych special issue: 70% failure rate; Many Labs 3: 88% failure rate).

 

Effect

Cited
by

Unsuccessful
replications*

Successful
replications*

Elderly
priming

3703

Pashler et al. (2009)

Cesario et al. (2007, Study 2)

Doyen et al. (2012, Study 1)

Doyen et al. (2012, Study 2)

Achievement
priming

1672

Harris et al. (2013, Study 1)

Harris et al. (2013, Study 2)

Deliberation-without-attention
effect

973

Acker (2008)

Calvillo & Penaloza
(2009)

Lassiter et al. (2009)

Newell et al. (2009)

Rey et al. (2009)

Thorsteinson & Withrow
(2009)

Nieuwenstein & van Rijn (2012, Study 1)

Nieuwenstein & van Rijn (2012, Study 2)

Behavioral-consequences-of
automatic-evaluation

943

Rotteveel et al. (2015, Study 1)

Rotteveel et al. (2015, Study 2)

Glucose-boosts-self-control
effect

853

Cesario & Corker (2010)

Astrologo et al. (2014)

Disgust
priming

713

Johnson et al. (2015)

Zhong et al. (2010, Study 2)

Physical
warmth promotes interpersonal warmth

693

Lynott et al. (2014, Study 1)

Lynott et al. (2014, Study 2)

Lynott et al. (2014, Study 3)

Money
priming

686

Tate (2009)

Grenier et al. (2012)

Rohrer et al. (2015, Study 1)

Rohrer et al. (2015, Study 2)

Rohrer et al. (2015, Study 3)

Intelligence
priming

676

Eder et al. (2001)

Shanks et al. (2013, Study 4)

Shanks et al. (2013, Study 5)

Shanks et al. (2013, Study 6)

Shanks et al. (2013, Study 8)

Fertility
facial-preferences effect

602

Harris (2011)

Macbeth
effect

578

Earp et al. (2014, Study 1)

Earp et al. (2014, Study 2)

Earp et al. (2014, Study 3)

Gamez et al. (2011, Study 2)

Gamez et al. (2011, Study 3)

Fayard et al. (2009, Study 1)

Pre-cognition

393

Wagenmakers et al. (2011)

Galak et al. (2012, Study 1)

Galak et al. (2012, Study 2)

Galak et al. (2012, Study 3)

Galak et al. (2012, Study 4)

Galak et al. (2012, Study 6)

Galak et al. (2012, Study 7)

Ritchie et al. (2012, Study 1)

Ritchie et al. (2012, Study 2)

Ritchie et al. (2012, Study 3)

Galak et al. (2012, Study 5)

 

Status-legitimacy
effect

344

Brandt (2013, Study 1)

Brandt (2013, Study 2)

Brandt (2013, Study 3)

Red-cognitive-impairment
effect

321

Steele et al. (2015, Study 1)

Steele et al. (2015, Study 2)

Steele et al. (2015, Study 3)

Steele et al. (2015, Study 4)

Power
posing

304

Ranehill et al. (2015)

Koch & Broughal
(2011)

Cleanliness
priming

278

Johnson et al. (2014a, Study 1)

Johnson et al. (2014a, Study 2)

Lee et al. (2013)

Johnson et al. (2014b)

Reduced
pro-sociality of high SES effect

258

Korndorfer et al. (2015, Study 1)

Korndorfer et al. (2015, Study 2)

Korndorfer et al. (2015, Study 3)

Korndorfer et al. (2015, Study 4)

Korndorfer et al. (2015, Study 5)

Korndorfer et al. (2015, Study 6)

Korndorfer et al. (2015, Study 7)

Korndorfer et al. (2015, Study 8)

Morling et al. (2014)

Social
distance priming

247

Pashler et al. (2012, Study 1)

Pashler et al. (2012, Study 2)

Johnson & Cesario
(2012, Study 1)

Johnson & Cesario
(2012, Study 2)

Sykes et al. (2012)

Color
on approach/avoidance

227

Steele (2013)

Steele (2014)

Social
warmth embodiment effect

117

Donnellan et al. (2015, Study 1)

Donnellan et al. (2015, Study 2)

Donnellan et al. (2015, Study 3)

Donnellan et al. (2015, Study 4)

Donnellan et al. (2015, Study 5)

Donnellan et al. (2015, Study 6)

Donnellan et al. (2015, Study 7)

Donnellan et al. (2015, Study 8)

Donnellan et al. (2015, Study 9)

Ferrell et al. (2014)

McDonald et al. (2015)

Red-boosts-attractiveness
effect

98

Banas et al. (2013)

Blech (2014)

Hesslinger et al. (2015)

Fertility
on voting

60

Harris & Mickes
(2014)

Process
model of AMP

55

Tobin & LeBel
(Study 1)

Tobin & LeBel
(Study 2)

Honesty
priming

47

Pashler et al. (2013, Study 1)

Pashler et al. (2013, Study 2)

Pashler et al. (2013, Study 3)

Modulation
of 1/f noise on WIT

45

Madurski & LeBel
(2015, Study 1)

Madurski & LeBel
(2015, Study 2)

Embodiment
of secrets

31

LeBel & Wilbur (2014, Study 1)

LeBel & Wilbur (2014, Study 2)

Perfecto, Moon, & Nelson (2012)

Time
is money effect

25

Connnors et al. (in press, Study 1)

Connnors et al. (in press, Study 2)

Heat
priming

23

McCarthy (2014, Study 1)

McCarthy (2014, Study 1)

Treating-prejudice-with-imagery
effect

22

McDonald et al. (2014, Study 1)

McDonald et al. (2014, Study 2)

Religion
priming

11

McCullough & Hone (2015)

Attachment-warmth
embodiment effect

10

LeBel & Campbell (2013, Study 1)

LeBel & Campbell (2013, Study 2)

A simpler and more intuitive publication bias index?

At this past SPSP, Uri Simonsohn gave a talk on new ways of thinking about statistical power. From this new perspective, you first determine how large a sample size you can afford for a particular project. Then, you can determine the minimum effect size that can reliably detected (i.e., 95% power) for that sample size (e.g., d_min = .73 can be reliably detected with n=50/cell). I believe that this approach is a much more productive way of thinking about power for several reasons, one being that it substantially enhances the interpretation of null results. For instance, you can conclude (assuming integrity of methods and measurement instruments) that the effect you’re studying is unlikely to be the size of the minimum effect size reliably detectable for your sample size (or else you would have detect it). That being said, it is still possible the effect exists but is much smaller in magnitude, which would require a much larger sample size to reliably detect.

In this post, I use the core ideas from this new approach to come up with a simpler and more intuitive way of gauging publication bias for extant empirical studies.

The idea is simple. If a study reports an observed effect size smaller than the minimum effect size reliably detectable for the sample size used, then the study likely suffers from publication bias and should be interpreted with caution. The further away the observed effect size is from the minimally detectable effect size, the larger the bias. Let’s look at some concrete examples.

Zhong & Liljenquist’s (2006) Study 1 on the “Macbeth effect” found a d=.53 using n=30/cell. At this sample size, however, only effect sizes as large as d=.95 (or greater) are reliably detectable with 95% power. On the other hand, Tversky & Kahneman’s (1981) Framing effect study found a d=1.13 using n=153/cell. At that sample size, effect sizes as small as d=.41 are reliably detectable. See Table below for other examples:
minimum effect size

The new bias index can be calculated as follows:  minimum effect size - bias-equation

(And note we’d want to calculate a 95% C.I. around the bias estimate, given that bias estimates should be more precise for larger Ns all else being equal.)

To shed more light on the value of this simpler publication bias index, in the near future I will calculate these for studies where replicability information exists and test empirically whether the index predicts lower likelihood of replication.