2012 Impact Factors – Evolutionary Psychology and Evolution & Human Behavior

The 2012 Impact Factors (IF) – one measure of the influence of a scholarly journal – have been released. Impact factors are controversial, and what they are measuring is richly debated. I have no strong position on this issue, though in general I applaud the proliferation of different metrics for evaluating journals.

In any case, for better or worse, the new 2012 IFs are out. Evolutionary Psychology, the journal that hosts this blog, leapt from last year’s 1.055 to 1.704, the highest the journal has ever achieved. The present Impact Factor puts Evolutionary Psychology in the neighborhood of Personality and Individual Differences (1.807), European Journal of Social Psychology (1.667), Sex Roles (1.531), and, another evolutionary journal, Human Nature (1.814). A nice neighborhood. Kudos to the editorial team, especially Todd Shackelford, who has worked tirelessly over the last several years to put the journal on the path that it is currently traveling.

Evolution_and_Human_Behavior_coverThe other journal I’ll mention is the official journal of the Human Behavior and Evolution Society, Evolution and Human Behavior, which increased from 3.113 in 2011 to 3.946, also the highest it has ever been.

One way to put this value in perspective is to place it among journals in social psychology. Set against the 60 journals listed in the Journal Citation Reports, Evolution and Human Behavior would rank 4th, behind Personality and Social Psychology Review, Advances in Experimental Social Psychology, and Journal of Personality and Social Psychology (4.877). In anthropology, E&HB would be third of the 83 listed, behind Journal of Peasant Studies (superstar paper: “Globalisation and the foreignisation of space: Seven processes driving the current global land grab” by A. Zoomers, cited 144 times according to Harzing’s) and the Journal of Human Evolution (4.094). (Current Anthropology is at 2.740, which surprised me because I thought it would higher.) Ranked against economics journals, E&HB would be 4th of the 332 listed by JCR. (The journal is grouped by Reuters under Biological Psychology, and sits second, below Behavioral and Brain Sciences with its whopping 18.571 Impact Factor.)

The bulk of the credit for these numbers goes, of course, to the authors of the papers. Still, I do think that it’s worthwhile to take a moment to note that the present quality of the journals, such as they are, owes a tremendous debt to the prior editors of the journals. I have some slight worry that some of the younger generation in the field might not know that Martin Daly and Margo Wilson edited Evolution and Human Behavior (starting before, in fact, it was called Evolution and Human Behavior) from 1997 to 2005, and are to be credited with setting the journal on its present trajectory. Steve Gaulin, Dan Fessler, and Martie Haselton are similarly no longer actively editing, but they each and all shepherded the journal through the 2000’s. The present group of editors stands on the shoulders of the efforts of all of these scholars who came before, and I, for one, am deeply grateful for all their hard work.

24. June 2013 by kurzbanepblog
Categories: Blog | 6 comments

Comments (6)

  1. You wrote:

    “I have no strong position on this issue”

    As a scientist, you ought to have a strong position. IFs are the equivalent of homeopathy, dowsing or astrology in science and there is enough empirical evidence to make a very compelling case for that, reviewed here:

    http://www.frontiersin.org/Human_Neuroscience/10.3389/fnhum.2013.00291/full

    In brief, IFs are negotiated, irreproducible and mathematically unsound. They also correlate *negatively* with several measures of article quality.

    For high-IF support of these conclusions, see here:

    http://bjoern.brembs.net/2013/06/everybody-already-knows-journal-rank-is-bunk/

    • Björn,

      I read your interesting FHN paper and some of the work you cited there. Thanks for sharing. I have a few comments to make:

      1. If I had to summarize the “big picture” of the findings you reported, I would say that there seems to be a trade-off between innovation and reliability-replicability, independent from methodological soundness (which is more or less the same across journals). High-ranking journals can be seen as “specializing” in novelty at the expense of reliability-replicability, whereas low-ranking journals follow the opposite pattern.
      This seems very reasonable and fits my own perception; however, in your interpretation of the findings you might be over-rating the value of replicability and downplaying that of novelty in science (note that novelty is not just an attribute of an empirical datasets; it may also depend on how the data are interpreted and used to develop or test a new theory). IF it were more difficult to come up with a “new and surprising” paper (theory+finding+interpretation) than with a “replicable and reliable” paper, and IF such novel papers had more potential to stimulate new ideas in the reader (even if they are more often wrong or inflated), THEN it would be rational for scientists as a whole to subjectively assign more prestige to high-ranking journals.
      The higher likelihood of retractions and “flukes” in high-ranking journals could then be seen as a calculated risk. Indeed, you might add risk to the equation and say that high-ranking journals specialize in novel, high-risk papers, whereas low-ranking journals specialize in replicable, low-risk papers. Again, it can be rational to assign more value to risky options iw what actually counts is not the average result but the really outstanding papers that permanently change the game in a discipline (which are hard to identify in advance).
      Mind you, I’m not arguing that you are wrong–just that you seem to take a very risk-averse stance toward scientific publication (at least in the linked paper); however, how scientist as a whole weigh novelty and risk versus reliability may explain the “puzzling” discrepancy between subjective evaluations of journals (which do correlate strongly with IF) and the story told by your empirical indicators. Moreover, there are often rational grounds for trading off reliability for novelty and risk; thus, subjective evaluations of journals may have more value than you seem to believe.

      2. Even if standing on the novelty-reliability dimension were the ONLY information conveyed by a journal’s rank, that would still be potentially useful information. An individual scientist is a busy machine, and has to make countless decisions every day concerning what papers to read and what to ignore. Depending on my current goals, I might want to focus on novel, thought-provoking papers versus reliable but less provocative ones. Also, I cannot afford to read the entire literature every week before I make my mind up. It is quite rational for me to rely on a journal’s IF as a quick decision heuristic.

      3. You report r^2 values between .1 and .3 for the correlation between IF and citations, then dismiss them as very low and “practically useless”. This seems an overly strong position to take, for two reasons. First, I guess the r^2s represent linear associations, but I would bet the actual relation is not a linear one. If you (rightly) criticize the IF for computing means on skewed distributions, you also have to admit that linear r^2s are likely to be artificially low if the relation between IF and citations is nonlinear.
      Second, r^2s between .1 and .3 amount to correlation coefficients between .3 and .6. These are not “useless” correlations, especially in a noisy and unpredictable environment such as the citation “market”. In binomial form (BESD), a r^2 of only .15 means that, by relying on a journal’s rank, I can improve my chances of reading a (future)high-impact paper from 50% to 70%. This doesn’t look useless to me, especially if you consider the cumulative predictive advantage over a long sequence of decisions (reading vs. not reading each individual paper I might read on a given day). And on top of that, the r^2 value is likely to be an underestimate of the true association if the latter is nonlinear.

      • Thank you so much for the detailed, thoughtful and well argued comment. When I first read it, I thought “could we really have committed such a blunder?”

        However, upon further thought, I feel like I should point out a few aspects of the data, which, I think, would strongly weaken your arguments, at least according to my understanding of the literature.

        1. Risk/reliability trade-off and being risk-averse. That one stung, lol! Excellent! I think, your description of hi-IF journals as hi-novelty/lo-reliability is quite accurate. I think there is a good analogy in newspapers: the gutter press or tabloids: high novelty and attention-grabbing reports, but often not quite accurate and of doubtful general relevance. If that were indeed the common perception, it would be accurate and I would probably have no quarrels with that. After all, if universities want to hire and fund people who publish in the equivalent of the The Sun or BILD or News of the World, that’s ok for me as long as that’s how it is also portrayed. In fact, a professional Nature editor once told me she had rejected a manuscript for Nature on ‘beer goggles’ because she *didn’t* want Nature to become a tabloid. The data show, that’s what they already are, you are quite right about that.
        The tabloid analogy goes even further than that: take the MMR scare: just as smears in tabloids can cost livelihoods and lives, so can incorrect publications in scientific tabloids: children have died from measels since Wakefield’s paper.
        In brief, I think your assessment is correct and if the scientific community were honest about it, then probably nobody would worry about the Wakefields, Stapels, Wangs, Schöns and all the other fraudsters: high risk – high gain! Personally, I think this would be a tough sell to the public and politicians, but why not?

        However, portraying the research published in these journals as ‘the best’ is simply not accurate. Moreover, as we already seem to be employing the people who are better at marketing their ‘discoveries’ than at actually making discoveries, one can expect the retraction rates to increase exponentially: it’s a feedback loop of like breeding like: more risk means more retractions. BTW, if one extrapolates the curve in Fig. 1a, one hits 100% retractions already in 2046. Given these trends, I think there is a place for risky results, but these risky results, as you point out, should be “permanently changing the game in a discipline” and not threaten to bring down the entire scientific enterprise as we know it today.

        What I didn’t understand is why one needs to assign that value in advance (and hence bestow positions and funds to the authors) rather than test them for accuracy? What’s the value for science in blasting out some potentially game changing ‘discoveries’ (like the MMR-scare) without first testing them thoroughly? It’s not like the people in the field don’t read their regular journals.

        2. Given that the high-profile papers are really rare even in the Stapel-Schön-Wang-Wakefield-GFAJ journals, that only a tiny fraction of these few articles will ever be in a field allowing you to actually read the papers (as opposed to their news service, which also incorporates news from other journals!!)combined with the high false positive and false negative rates (even without retractions and such! see below), the error you introduce by random selection might not be noticed until years after you changed to dice-throwing, if at all. Given many of these journals appear weekly, signal-to-noise in these general journals is orders of magnitude below topic-oriented journals. But it wouldn’t hurt to actually calculate this!

        3. Related to 2: I agree about the linear vs. non-linear issue. One would need to dig into the vast literature on IF vs. citations, I’m sure many of these articles are discussing this. Of the top of my head, I can’t point you to one, I’m afraid.
        However, the low r^2s are not the only reason to argue that the correlations are useless. There are (2 I think) studies we cite on classification of papers and how often papers are wrongly placed in higher tier journals (false positives) and how often they are placed in too low a journal (false negatives). Those values seemed excessively high (approaching 50% in some cases, IIRC), but it would of course not be a bad idea to try and device some real-world examples of how the odds are of hitting ‘hi-profile’ papers with various reading strategies.

        Finally, evaluation for reading is of course only one side of the story. In our opinion, even if there were the modest benefits you claim (which I think may not be there, but I would be persuaded otherwise by data), the pernicious incentives provided by the hire-and-fire system we have built around journal rank, annihilate any perceivable benefits this current system might be found to have (at least as long as the current data are even in the ballpark of indicating the potential size of these effects).

        Thus, in conclusion, I would say there is some merit to your points, but given that reading is not the only function of journal rank, the data on the cost side still by far outweigh the benefit side. Moreover, any of these effects would be easy to implement in a system where the pernicious incentives have been minimized as opposed to maximized, as in the current status quo.

        • Thanks for your response! That of incentives is another big issue, and I think you have some really good points to make, so I won’t go there. My argument revolves around the correspondence between IF rankings and scientists’ subjective perception of journal quality/prestige. If rankings seem useless based on a certain set of data, either (a) scientists as a group are deluded and irrational, or (b) they are using rational criteria that are not well captured by the data (or by the theory used to interpret the data).

          Since your paper is being widely circulated and you will likely publish again on this topic, I’m trying to make you see things from another angle and consider a fuller range of explanations for the current state of things. I think there is more rationality in the system than you acknowledge; I’m sure there are better alternatives, but they will only work in the real world if they will satisfy people’s actual motives and decision-making processes. Getting those wrong is a recipe for well-intentioned failure.

          To rephrase my previous comments, my first point is that accuracy (in the sense of replicability) is not the only valuable aspect of a scientific paper. I’m not talking about frauds or fabricated data (of course I don’t condone that)–I’m talking about real surprising data (that may or may not replicate well) and brilliant innovative ideas (that may or may not pan out in the long run). I often read papers that I think are wrong or inaccurate in some respect, but open up new ways of thinking, suggest new research paradigms, and so on. Those are good papers to me–in fact, more valuable than a string of well-done studies that perform small variations on the usual association between X and Y (not that the latter are worthless, but still). As long as you only focus on accuracy-replicability, you will get a very distorted picture of what people value in a scientific paper and why.

          My second point is that the publication game is (a) very noisy and (b) ruled by outlier cases. In such conditions, focusing on average measures of performance, r^2s, and the like may also distort your thinking. For example, virtually every successful scientist only has a few high-impact publications, some of moderate impact, and a majority of comparatively low-impact and seldom cited ones. In a sense, your reputation revolves around a handful of outlier cases. This is not the result of some perverse incentive system in academia, as the exact same pattern occurs in music, literature, and so on. Also, scientists are rather poor at predicting which of their publications will have a large impact and which will not (see Dean Simonton’s wonderful books, “origins of genius” and “creativity in science”).

          In these conditions, averages may not be very informative. For example, given that maybe 1 in 10 of my publications will have a real impact on the community (if I’m lucky), something that increases my chances of a “hit” by just 10% (e.g., publishing in journal X) is still worth a big investment. Also, the rules of the game may be such that only small improvements over chance are possible to begin with. If so, r^2s and similar measures may be deeply misleading, as no causal factor can possibly explain more than a few percents of variance. Still, small causal factors can have a cumulative impact as the publishing cycle is repeated over and over. Another example: when I scan the new issue of a high-ranking publication in my area(s), I find maybe 1 or 2 papers out of 20 that seem worth downloading. When I scan the low-ranking journals, the modal number is 0. The mean difference is small, but it’s rational for me to scan the high-ranking journals first. Or, I like the idea of PLoS ONE as much as the next guy, but every time I get a new alert I get a chill–I know I will have to sift through pages, and pages, and pages of mostly uninteresting publications before I find something I care about. As a result, I often procrastinate, and my PLoS ONE alerts sit in my inbox for a long time before I open them (when I do). The general point is, the value of a journal does not lie just in the average quality of the papers, but also in the quality of the outliers and in the ability to organize and present information in a time-saving, cognitively effective way.

          I guess I’m done with my reply–thanks for the really interesting discussion, and good luck with your work!

          • Many thanks for your -again- very thoughtful and largely spot on comment. Just a few concluding remarks:

            Scientists need not be a) deluded or b) their data not captured – although both is certainly possible 🙂

            Journal rank can be explained by some simple psychological phenomena that nobody is immune against (but data helps). For one, there is the effect of artificial scarcity (e.g., see here:

            http://blogarchive.brembs.net/comment-n606.html )

            Together with confirmation bias supported by incomplete sampling, we constantly reinforce the idea that there must be something to journal rank, at least grossly inflating any effect there may hypothetically be.

            I appreciate you pointing out the potential rational components in our current system – I will pay more attention to these in the future and emphasize more that none of these are threatened by moving to a more state-of-the-art scholarly communication system. Clearly, as always with major reforms, the potential for well-intentioned failure are huge, no question there.

            Perhaps I didn’t emphasize this enough in my previous reply: I really do appreciate the validity of the two points your rephrase here: landmark (nonlinear) discoveries need to be widely circulated, obviously, and easily discovered. The main point of our paper was that today it would be technically easy to do that with orders of magnitude better statistics than today – but journal rank prevents us from implementing such common technology. Even if the effects you mention would one day be detected, they’d be still minuscule compared to the effects we could get out of a properly designed communication system. Without that, the pernicious incentives already more than cancel out any potential benefits one might discover with more sophisticated analyses in the future.

            WRT your last example: for me there hardly is ever anything worth reading more than the abstract of in the GlamMagz, while there often are papers in lo-IF journals that I need to read abstract-references. So for me, it’s the opposite than for you: most of the stuff in GlamMagz is superficially interesting – it suffices to read the headlines to know what’s going on (e.g., Higgs Boson discovered!), while the actually important, relevant and game-changing work virtually never appears in these journals.
            For instance, in one of my fields (Drosophila learning and memory), it is easier to get a new learning mechanism published in Nature that is specific to Drosophila for a paradigm that everybody knows, but if a new learning mechanism is discovered in a less well-known behavior, but where there is evidence it might be conserved among all bilaterians (i.e. a more general finding), it will be published in other journals. Confirming this anecdote is another anecdote from personal experience: our Science paper in Aplysia is the single paper of all my experimental papers where there isn’t a single novel finding presented: only confirmation of other, individual experiments collected in a single paper.
            Thus, from my perspective, it’s exactly the opposite from yours, not only in reading, but also in terms of publishing. If the experience of individual researchers varies this widely, it’s not surprising that the statistics over large numbers show little actual effect. Consequently, some researchers will say: oh no, you just don’t get it, while others will say: I’ve always suspected there was something wrong with journal rank. In the end, what matters is not subjective impressions from individuals, but tangible effects that we can build evidence-based policies on – but there only few scientists will disagree, I hope.

            Many thanks again for your very thought-provoking and well-founded comments! I’ll make sure I’ll keep them in mind. Good luck with your work, too.

          • Thank you. BTW, I realize I also didn’t make one point clear: when I say high-ranking I mostly mean high-ranking *within* my areas of interest. Actually, it may be more accurate to say that I experience an inverted-U relation between ranking and relevance, peaking at high-ranking specialty (e.g., Evolution) or semi-specialty (e.g., BBS, Proceedings R Soc B) journals rather than Science, Nature, and PNAS. Maybe this has to do with my theoretical bent; maybe the “GlamMagz” could be analyzed separately from the rest. In any event, it was instructive to exchange persepectives! M

Skip to toolbar