The quality of reviews
The peer review system is there to guarantee a high quality standard for scientific papers. If all scientists were rational, selfless and fully objective beings with unlimited amounts of time, this system would be a good idea to find errors before publishing. However, if you put real scientists into the equation, scientists that are normal humans, your get a low quality of review and a system that is prone to power abuse.Writing a good review can take a few days and brings your career nothing, even if it may bring some personal joy to dive deep into the topic and learn from it. Nowadays a scientist is supposed to publish many articles, and acquire projects (and possibly give good lectures), and all other important duties, such as reviewing, are not measured or valued by our funders. At the same time a scientist does feel pressure to do reviews. You do not want to annoy the editors of the journals you would like to publish in yourself later on. The result of these conflicting interests is that in many cases the review is done in haste and often also well after the deadline.
Bad review examples
One of my first reviews was easy. The reviewer wanted to have 5 references to his articles, one of which was vaguely appropriate, I gave him three references, which he apparently found enough and the paper was accepted. This happens so often that senior scientists almost regard it as normal.Another example of power abuse is a reviewer that rejected an article of mine, only to try to publish a similar article himself. This copycat had the bad luck that he got a reviewer who already knew our work from a conference and consequently rejected his article as not new and told me about it.
Another typical problem is that if you criticize the work of others, the editors somehow feel obliged to send the manuscript to the scientist that is criticized. I do not know why editors think scientists are so objective and super-human, that they have no problem with a public exposure of their errors. In the two cases I know, this mistake of the editor has led to a request by the reviewer to expand and clarify the error that was made, but you can read between the lines that the reviewer wants you to drop the subject. In one case, the clarification, of course, led to a rejection without much words. In the second case the reviewer gave up after a long battle that was much work.
I once attended a colleague on an error in his article. He replied he knew it was wrong, but the reviewer wanted him to write it, and it was only a minor point. Publish or perish. Who is the first to throw a stone?
Anyone who knows my work on surrogate cloud fields, knows that one of their strong points is that the cloud water distribution is fully flexible, and thus can be adapted to a measurement. Still, one reviewer asked me why I assumed that the distribution is Gaussian; commenting on a part where I explained a detail of another algorithm, apparently the only section he had read. Also by most of his other comments, an expert could see that he had not read much. Still the reviewer rejected that article, as he had not found anything new in it. A clear example of a review made in too much haste, well after the deadline. One would expect that this lack of time makes it more difficult for more creative research to get published. An article presenting a new idea or a new philosophy takes more time to understand.
Reading the literature it is also clear that well written, i.e. articles in good English, and articles written by famous scientist most often contain (small) errors. Apparently, the human reviewers tend to judge the outside (language) and/or trust the good name of famous researchers, and invest less time in the review. However, I also heard of a case where an article of a well-known researcher was rejected by at least one reviewer and published after a phone call from the author to the editor, probably as the editor was keen on keeping good relations with the author and an article by a famous researcher that would give his journal a better citation index.
Time
Not only is the quality of the reviews not always optimal, the review system also takes a lot of time. The journal sets the reviewers a deadline. For many journals I know, this is about two months. For many reviewers, being busy people, this means that the first two months they do nothing; they wait until the editor starts asking. In practise you always have at least one such reviewer. In total, the entire review can easily take half a year to a year. And before the article is published, it takes one and a half to two years in meteorology. This slows down scientific progress.Peer review reform
Below, I present a few ideas that will hopefully lead to a better quality review. I must admit that I do not expect that the journals will implement these ideas as they would require additional work. Anyway, the problem is likely more fundamental and these small changes will thus probably not be sufficient. Therefore, I am dreaming of a new fully system for judging the quality of scientific works and simply forget about journals. Anyone can publish his work in this internet age, the only thing we need is the ability to trust articles on subjects where we are not specialists. Thus, we need to build networks of trust, which social networking site do very well nowadays.Editorial policy
Starting with a positive view on the human reviewers, I feel it is important to involve reviewers more. This could make some of them more dedicated. The editor should tell the reviewers what the other reviewers thought, and what the final decision is. Especially in case of major revisions or rejections, it would be good to give reviewers each other's reviews so that they can comment upon it. After spending much time on reviewing a manuscript, you become involved in the topic and would like to know what happens with the article.An important point is not to send manuscripts to people that are criticized, but to independent experts. If needed with the special request to judge the appropriateness of the criticism. It is naive to expect that scientists are so rational, unemotional and uninterested in their reputation, that they will trouble-freely accept criticism.
Furthermore, the editor should communicate the importance of deadlines and make them shorter even if this results in more people rejecting to do reviews and thus more work for the editor.
Black list of reviewers
Some reviewers are also simply incompetent or immoral. To get rid of them, I would suggest to make a list with reviewers and how much complaints there are about them and how much time they needed. This list can be used to select reviewers and to judge their reviews. Preferably, such a list should be a joint list of all journals in a certain area so that one obtains significant results much faster. The reviewers should have access to their own listing and be able to protest against it. Although, most reviewers will probably be happy with a high position, as it would mean less reviews.Open review
Many EGU Journals use an open review system. Having read a number of these reviews, I feel that these reviews are on average of a higher quality as the ones in the closed anonymous review system. If you submit an article to such a journal, your manuscript is published in a discussion journal in internet, so that you can claim ownership. Then the journal seeks two anonymous reviewers and all other readers can comment on the manuscript and on the anonymous reviews.This system involves the reviewer much more; they can read the comments by the other reviewer and readers. And who would like be publicly accused of doing a bad review? Even if the review is anonymous, that would be bad for your self-esteem.
I can advise all scientists that want to stay in the peer review system, to publish in such journals. In the internet age and with the enormous number of articles that are written, it is not very important anymore in which journal you publish. What does help spread your ideas, is the ability to freely download articles, which these open-review journals allow for as well (they are also part of the open-publishing movement). If you have the choice between two similar articles and for one you have to go to the library to get bad quality copies and the other one you can download and print in high quality. Which article would you read?
An added benefit of open-review could be that also after the official publication date, it would be possible to comment on the article informally, without having to write a commenting article. This way an article could be starting point of an interesting scientific debate.
I hope this essay will start a discussion in the scientific community, that the victims of bad reviews realise that they are not alone and will protest, and that in the end the review system will be fairer and quality more important. I am afraid that instead the messenger will be blamed for ruining the good name of science. I am also afraid that good scientists will see this essay as a personal attack. Thus I would like to emphasis that there are still many good reviewers around, and that it is the way science is organised that puts a strong pressure on people to do bad reviews. As long as jobs and projects are not based upon someone's contribution to science, but based on someone's number of articles, this pressure will not go away.
This post was first published on my homepage on 10 December 2005.
[UPDATE 2014. This post was written almost a decade ago. Since this time my experiences with peer review have become more positive. Maybe I have learned to play the system better, maybe it helps that I nowadays prefer to publish in European journals, where scientists still have more time to do reviews and the publish-or-perish mentality is not as strong. Furthermore, I have now more experience as a reviewer myself. Also being older, I have more publications and the delay of a single publication is not as important any more as it was when I was just beginning. The power abuses mentioned can only delay publication, a legitimate scientific work will always find a journal.
The suggested improvements still sounds like a good idea to me. Maybe I am biased due to my change in role, but I no longer think that we should abolish peer review before publication. In fact the peer review can help a young scientist or an outside idea to gain the initial credibility that can entice scientists to invest the time to read the article. What I did not think of above is that the problem is not getting published, but is getting read by the right people.
While, the post does not provide direct arguments, the under-title suggests that I was also against the anonymity of the reviewers. If I were, I have changed my mind. Just like someone does not like a manuscript that critiques his work (and is tempted to reject it if this person is asked to review the manuscript ), also the author does not like it when reviewers critique or reject their work. Without the anonymity the reviews likely become much more positive, but the science would not get better. Rejecting a manuscript would become a liability and the reviewer would have to fear retaliation for a manuscript or research proposal where the roles are reversed. A tenured professor (the main source of power abuse mentioned above) hardly has to fear that happening when he reviews a manuscript of a young scientist, but in other combinations that danger is clearly there.
An improvement would be when both sides are anonymous. Typically you can guess from which group the authors come, but not always and guessing whether it is the professor or the PhD student can be difficult.
The situation is different from other professionals making judgements and signing them, such as medical doctors or architects. In this case making the right decision does not have negative consequences for your career. In case of peer review that is possible. Furthermore, science is working on the edges of what we understand. In this fuzzy situation many manuscripts, but also many review decisions, will be wrong. Whereas for doctors or architects there are clear standards; if they stick to them, they are save. Finally, scientists volunteer to do reviews, for other professionals making judgements is their job. (I hope we will have a system one day where reviewers are rewarded for writing good reviews and editors for making good decisions.)
]
Related reading
Peer review helps fringe ideas gain credibilityThree cheers for gatekeeping
The value of peer review for science and the press
No comments:
Post a Comment
Comments are welcome, but comments without arguments may be deleted. Please try to remain on topic. (See also moderation page.)
I read every comment before publishing it. Spam comments are useless.
This comment box can be stretched for more space.