The Impact Factor Game
I didn’t intend to talk about Impact Factors so early in this blog but my colleagues on PLoS Medicine have written such a good editorial on the subject, The Impact Factor Game, that I couldn’t let it pass without mention.
It doesn’t take a great deal of thought to see why the ‘worth’ of a paper isn’t well assessed by the Impact Factor of the journal in which it is published. Impact Factors are essentially the average number of citations for papers in a particular journal. Problem is citations aren’t normally distributed across those papers making the power of that average to predict the likely citations of an individual paper very low. As a rule of thumb 80% of a journals impact factor is determined by 20% of the papers published.
Anyway that isn’t the point of the editorial. Rather it looks at the vagaries of getting an Impact Factor at a journal level, and the realisation that Impact Factor isn’t some objective measure but is open to a large degree of interpretation. At the moment it is Thompson Scientific whose interpretation is relied upon by scientists and editors alike:
During the course of our discussions with Thompson Scientific, PLoS Medicine’s potential impact factor – based on the same articles published in the same year – seesawed between as much as 11 and less than 3.
You can imagine the amount of hair pulling and rending of clothes that caused!
With a journal like PLoS ONE Impact Factor will be irrelevant. It will be far too broad and I don’t intend to get into the game. However there will still be a need to provide guidance as to the ‘potential worth’ of papers published. I’m personally very interested in the way that the algorithms behind Google page ranking are being applied to citation analysis both at the journal level with the concept of the Y-factor and at the individual paper level as in the recent paper Finding Scientific Gems with Google.
Along side such attempts at an objective ranking the opinion of scientists shouldn’t be forgotten. That’s why with PLoS ONE we are also keen to explore ‘user ranking’ of papers. This isn’t a new concept, it is the whole raison d’etre of Faculty of 1000 and new sites such as Biowizard also have this feature. The difference with PLoS ONE is that ranking will be open to all and an integral part of the post publication peer review of papers.
Different ways of assessing papers tell you different things. Isn’t it better to have a diversity of measures rather than rely on a single, over-used and over-interpreted metric?
Given that the impact factor is a shifty number, it is strange to me that it is still quoted all over the PLoS Biology pages. If it weren’t a favourable number (closer to 3, as could be possible for PLoS Medicine), would it still get prominent placement on the site? At the very least, I think any reference to the impact factor of PLoS journals should link to that PLoS Medicine article.
The impact factor is so successful since it’s very simple and its definition reads so reasonable.
I always wondered about what papers do get included in the calculation, and indeed I think that the community should get transparent information on this subject. That is, ISI should explicitly deliver explanations on what papers are included for the calculation of the impact factor of each and every journal.
Even in this case, however, the impact factor remains a poor measure of the importance of an individual paper (while I feel that it could be a pretty good indication of the impact of a journal, provided it’s calculated in a fully transparent, homogeneous way).
Thus, the possibility of getting direct community feedback on individual papers is a major plus of the open access 2.0 project.
First, I agree that we need ways to measure scientific performance, but I am convinced that a fair future for measuring such scientific impact is only possible when a non-profit institution is responsible of developing and maintaining an open access information web portal including specialized and flexible ranking tools services, based on clear rules whether to include or not articles, and how a scientific article is defined. The whole situation is certainly too subjective. A scientific paper appearing in an important newspaper could exert a relatively strong impact in society, however e.g. see DOI: 10.1371/journal.pbio.0040055 “[W]hile I’m proud of the work, it’s certainly a disproportionate amount of attention given how many other interesting things there are in science.â€
Sadly such ideal scenario mentioned above is contradictory to our current trend, where ISI ranking is heavily influencing more and more the decisions inside the whole scientific community worldwide. Anyhow and after all the problem is that we have this need, the one of an impact factor to compare our peers and we have no alternatives, until now.
Could be this another goal for PLoS? May be yes.
This (or rather last) year’s journal Impact Factors have been released today. Just for your information PLoS Biology’s is 14.67 and PLoS Medicine’s is 8.39.
Good to see that PLoS Biology articles are getting cited!
Just wanted to follow up on comments about BioWizard and their open-access post-publication review of papers. As mentioned, they do contain new ranking features, but in addition the site is already freely open to everyone, and scientists can actually discuss and comment on any paper ever published and located in the PubMed database, that’s over 16 million articles!