What We Have Here is a Failure to Replicate
On Monday of this past week, Hal Pashler gave a talk as part of the Psychology Department’s colloquium series at my home institution, the University of Pennsylvania. His talk focused on the issue of replication in certain areas of experimental psychology. Some readers might recall something of a stir in the blogosphere when Doyen et al. reported a failure to replicate work that looked at whether priming people with words related to old age, such as “bingo,” and “Florida,” caused people to walk slower than those not primed with such words. Pashler discussed some work that he and his colleagues also published in PLoS reporting their attempts to replicate related studies showing that subjects who plotted points closer together experienced feelings of greater closeness to their families relative to subjects who plotted points further apart. I’ve put the results of the closeness study here in the Figure below. In his talk, Pashler discussed a number of other attempts to reproduce results of this general type, all with the same result: a failure to replicate.
Figure from Pashler et al. (2012). Ratings of closeness by condition.
The topic of replications has been discussed at some length, and I’m not in a good position to contribute anything substantive to this discussion, but I thought I would spend a few moments musing about the topic for a few reasons. First, there’s a new paper (paywall) by Ioannidis & Doucouliagos “What’s To Know About The Credibility Of Empirical Economics?” Holding aside their rather dismal evaluation of the dismal science — “the credibility of the economics literature is likely to be modest or even low” – I particularly liked the way the authors quite dryly expressed the general problem: “Replication is a public good and hence prone to market failure.” The footnote this with a reference to Dewald et al. (1986), who wrote: “A single researcher faces high costs in time and money from undertaking replication of a study and finds no ready marketplace which correctly prices the social and individual value of the good.”
Pashler had a great slide, a picture of a passage from a second grade textbook, telling the (young) reader that a cornerstone of science is replication. Given how rarely research is, in fact, replicated in many areas of science, the point is well taken. Which is not to say that there aren’t efforts being made to try to address the problem, including, for instance, the Reproducibility Project and Psych FileDrawer.
Discussions of why replications aren’t more common – including Pashler’s remarks – focus extensively (but not exclusively) on incentives. If a researcher attempts to do an exact replication of published work, there are two possible results. If the result replicates successfully, it is likely to be difficult to publish because journals tend not to publish replications, though this is changing. Last month, for example, Bobbie Spellman announced an initiative at her journal, Perspectives on Psychological Science, providing an interesting mechanism for publishing replications. Other journals are proving more receptive to publishing replications – and failures to replicate – which will probably have some beneficial effect. In any case, my guess, though I don’t know, is that replications of results are cited relatively infrequently, especially compared to the original results. Publishing failures to replicate is likely no easier than publishing successes.
The issue of incentives does not, of course, end with authors. One issue that the editorial team at Evolution and Human Behavior is discussing is what our policy ought to be in this regard. While I myself feel that the sort of Registered Reports that PoPS is soliciting have tremendous value, what will the effect be on the journal? There is little use denying that in the present era, journals – and their editors – are judged on quantitative metrics, especially citation counts. To the extent that replications, successful or not, draw fewer citations than new research, publishing replications entails a cost to the journal, exactly along the lines of the Dewald et al. quotation above: publishing such papers is enduring a cost to produce a public good.
If its’ true that publishing replications reduces the infamous impact factor – as well as other metrics – authors are affected as well. At many institutions, departments and personnel committees use metrics such as impact factor to evaluate the quality of the journal that candidates up for promotion are publishing in. Would contributors to particular journals be willing to pay the price of the reduced impact factor to support a policy of publishing replications?
This is not, exactly, a rhetorical question. I’m interested in the question of whether members of the evolutionary psychology community believe that E&HB should encourage/tolerate/permit the publication of replications and failures to replicate. (Please feel free to contact me offline. No need to make your thoughts public unless you want to.) I should note a couple of points. First, the journal doesn’t receive many replications, successful or otherwise. Second, I recently green-lighted a paper that was as close to a replication as one can do, given that the study was executed in a very different context from the initial study. So, there is a sense in which the journal is already in the business of publishing replications. Should it be?
Citations
Dewald, W.G., Thursby, J.G. and Anderson, R.G. (1986) Replication in empirical economics. The Journal of Money, Credit and Banking Project. American Economic Review 76: 587–603.
Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: it’s all in the mind, but whose mind?. PLoS One, 7(1), e29081.
Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PloS one, 7(8), e42510.
Ioannidis, J., & Doucouliagos, C. (2013). What’s to know about the credibility of empirical economics? Journal of Economic Surveys.
Pingback: What We Have Here is a Failure to Replicate | E...