A few days ago, Rolf Zwaan wrote an interesting post about “replicating down” vs. “replicating up”, which he conceptualized as decreasing vs. increasing our confidence in an effect reported in an original paper. I love this distinction and definitely agree that we need to see a lot more “replicating up” type replication efforts and that editors of prominent journals should publish results from such “replication up” replication efforts.
In this blog post, I’m going to tell a story that embodies a different kind of “replicating up” and contend that “replicating down” can lead to “replicating up” in highly constructive ways for our science.
Here is the story.
We executed two high-powered pre-registered direct replication attempts of an effect where we bent over backwards (a la Feynman) in collaboration with the original authors to duplicate as closely as possible all methodological details of the original study. However, we couldn’t get the effect. So we submitted our results to the journal that originally published the results (trying for the Pottery Barn Rule), but it was rejected for not making a sufficiently substantial theoretical contribution. The editor argued that for publication we needed to provide the conditions under which the effect *does* occur.1 In a weird twist of events, one of the reviewers — who was one of the original authors on the paper — reported in their review that they had since “discovered” a moderator variable for the effect in question. The action editor suggested we “combine forces” and consider re-submitting to the journal. Indeed a few days later, I received an email from “Reviewer #1” offering that we combine forces and submit a combined paper with our null replication results and their moderator evidence. I graciously declined the offer instead asking for the methodological details to attempt to independently replicate their new “moderator effect”. Suddenly, the researcher’s tone changed communicating that they hadn’t “yet pinned down the effect”, but would email me with the details as soon as they had them. That email never came.
Fast forward six months later. Out of the blue, an independent team emailed me indicating that they also failed to replicate the original results in question in an even higher-powered design. However, in yet another weird twist of events, their replication results spoke directly to the moderator question at hand, seriously calling into question the so-called “moderator effect” explanation of our failed replication results. I emailed the original author to see if there were any developments regarding their new “moderator effect” given that I was made aware of new evidence calling into question their moderator effect explanation of our failed replication attempts.
They replied saying that since we last communicated, they have realized that the operationalization of their target manipulation was overly noisy and that they have since “substantially improved” it to make it more precise.
What music to my ears that was to hear.
And this is what I mean by saying that “replicating down” can lead to “replicating up”! Our “replicating down” eventually led to a “replicating up” situation by getting the original researchers to improve their methodology in studying their phenomenon of interest.
Take-home message: We definitely need more “replicating up” situations, but that “replicating down” can lead to “replicating up” and that this is very healthy for our science!
Last thing: my story fits very well with the Feynman-inspired name of my blog “Prove Yourself Wrong”, which is that by proving yourself — or others — wrong, scientific progress is achieved!
“We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.” (Richard Feynman)
1. This is ludicrous; it would be as if the Journal of Electroanalytical Chemistry — where Pons & Fleischmann’s (1989) published their now discredited cold fusion findings — would have demanded that independent replicators provide the “conditions” under which cold fusion *can* be observed!