The value of peer review for science and the press

The value of peer review keeps on producing heated debates. An interesting example was the weekend that physics professor Richard Muller wrote an op-ed in the New York Times. Some claim that Anthony Watts halted his blog for 2 days and released a scientific manuscript and an accompanying press release on the same weekend to steal attention away from Mullers op-ed. Both the op-ed and the press release were about scientific claims that had not passed peer review. Thus the Washington Post asked: Is it okay to seek publicity for a work that is not peer reviewed?

Watts et al. manuscript The eventful weekend at end of July 2012, resulted in two worthwhile blog post in the New York Times (Andrew C. Revkin at dotEarth) and the Washington Post (Jason Samenow). The manuscript was clearly released prematurely and had serious methodological problems. A few days after the press release and the blog reviews, Anthony Watts still wrote: "I’m hoping to post up a revised draft, addressing many of those comments and corrections in the next day or two." And he opened a "work page" for the manuscript, which is so quiet you can hear crickets. Just when no one expected it any more, the zombie manuscript came back from the undead; this March Watts wrote about this manuscript: "we are preparing a paper for submission". I am not a native speaking. May I ask, if you write "we are preparing", that indicates an ongoing action, right? Is there any lower limit on the intensity of this action?

The other side of the question about seeking the press before peer review is should a journalist only write about peer-reviewed studies? Further questions that came up since are: Is it unscientific to cite non-reviewed studies? Should the IPCC limit itself to reviewing only the peer-reviewed literature? Is peer review gate keeping? Is peer review necessary?

As often the context is important. What the value of peer review is, depends on who you are? An expert or not, a journalist or a newspaper reader? Another important part of the context is how controversial the finding is.

The Value of Peer Review for Science

Peer review gives an article credibility. As such peer review is "just" a filter, it does not guarantee that an article is right. Many peer-reviewed articles contain errors, many ideas outside of the peer-reviewed literature are worthwhile. However, on average the quality of peer-reviewed work is better. Thus peer-reviewed work is more likely worthy of your attention.

If you are a scientist and an idea/study is about something you are knowledgeable about there is no reason to limit yourself exclusively to peer-reviewed articles, but it is smart to prefer them. A scientist will only use peer review to preselect, because you simply cannot read and check everything. Life is short and attention a very limited resource. I also see no problem in citing studies that are not peer-reviewed, whether scientific reports or conference contributions. I do feel that by citing such studies, you give them some of your reputation, you become partially a reviewer and should read them as careful as a reviewer would.

Peer review is far from perfect. It is not intended to and cannot prevent fraud. Some bad papers will get through and some good ones will be rejected. This can be annoying for the scientists involved, but given that peer review is just a filter, it is not that bad for science in general. It is only since the second world war that peer review has become the standard. We also had scientific progress before that time.

Having to read a few bad papers is no tragedy and will not slow scientific progress much. Recently, Hans-Joachim Lüdecke, the press officer of EIKE, a German blog that tries to challenge the quality of WUWT, published a paper in Climate of the Past. If after reading the abstract, you do not Google the title to learn about the mistakes, you can only blame yourself. A special recent case is the paper written by Alan Carlin (Wikipedia | his blog) for a special Issue with guest editor Alan Carlin(!). It is a special case, because by hiding the paper in the International Journal of Environmental Research and Public Health, (almost) no climate economist will have to waste his time reading it. Thus the peer review filter did do his job, albeit indirectly.

Also scientists manage to get bad studies published. I would say that it is noticeable that articles by well-known scientists more often contain little errors. And one famous scientist proudly told me that when one of his articles was rejected, he just called the editor to ask whether his journal would like to receive future well-cited manuscripts from him or not? The rejected paper was published without any changes. In case of well-known scientists you could even argue that this is okay; reviews do not always improve papers and if the articles are really bad, they mainly hurt their own reputations. The little errors are probably not always due to power abuse. I have the impression that stylistically well-written articles also contain more errors. The good name or the good prose may simply make the reviewer less critical.

The reverse, good studies that are not published, is more problematic. One problem is that if you criticise a previous study, some editors send the manuscript to the author of the criticised study and treat the reviewers recommendations to completely revise the manuscript or reject it as if it came from a neutral, disinterested person. Fortunately not all journals do. In the end, a good study will find its place in the literature. There are so many journals and editors that "gate keeping" of good articles is impossible, such problems mainly cause delays. For the scientists involved this is very annoying and may slow down their career. Thus this is something that should not happen and something we should fight, but it is no foundation for the conspiracy theories of the climate "sceptics".

Concluding, bad papers in the scientific literature are a minor problem for science as long as their number is limited (and people still take the time to blog about them or write a formal comment). Whereas, a good idea that is not published is a big loss to science.

The main effect of bad papers is outside of science, if they confuse the population about the current state of science. Politicians like immutable truths and sometimes call for water-tight review systems. That is impossible, would add so much overhead that it would slow down scientific progress and make it very difficult to publish unorthodox ideas, which are typically weak in the beginning. As a scientist, I would thus prefer to have a peer review filter that errs on the side of publishing a few articles too much.

Donna LaFramboise, thinks that a study that is not peer-reviewed should be disregarded by the Intergovernmental Panel on Climate Change, IPCC. After the previous discussion, I hope it is clear that I could not care less whether they cite peer-reviewed literature or non-reviewed works. Every IPCC author writes about those topics where he is knowledgeable and should be able to judge what is worth citing. The citation policy of IPCC sounds fine to me.

Peer Review & Publicity

If you are not knowledgeable and have little time it is best to limit yourself to peer-reviewed studies. As journalists are interested in spectacular new ideas, the scientific articles headlining the science sections are already very likely to be found to be (partially) wrong later on. (As an aside, I would advocate journalists not to write about single articles, but about new ideas that are supported by a number of articles, as this reduces the likelihood of reporting erroneous ideas.) Furthermore, as the general public is not in the position to judge the value of a new paper, also scientists should show restrained in seeking publicity for non-reviewed studies.

Because it is a matter of credibility, not only peer review is important. Also the reputation of the scientist and how controversial the matter determine whether the time is ripe for publicity. I do not think that many people would object to the press conferences on the discovery of the Higgs boson, although these results were not published yet. However, the group is highly reputable in this field, the finding is a confirmation of an well-established theory and the experimental set-up has been vetted before building the enormously expensive instruments. Furthermore, the topic is of high public interest. Trying to delay its publication in the mass media while waiting for a peer-reviewed article would have been impossible.

In the same vein as the Higgs story, I see no problem with the op-ed of Richard Muller, mentioned in the introduction. Yes, it was before peer review. But if a physics professor confirms what climatology already knows for decades, you cannot see such a story as especially controversial. The press attention was consequently also not really about the science, but a human interest story of a self-proclaimed climate-sceptic, funded by the libertarian Koch brothers, being convinced by the data that global warming is real.

From private to public communication

A modern "problem" is that the distinction between communication between colleagues and the public is becoming less clear. From a private mail to one or multiple colleagues, to a small or large e-mail distribution list, from your personal homepage, to a blog post or a guest post at a well-known blog and from a working group seminar to a conference presentation. Where does publicity start?

As a consequence it can be impossible to control an interesting story and stories about non-reviewed studies are likely to become more common. Our Italian colleagues that had measured speeds faster than the speed of light had no other option as going to the press after all the attention their results got in the blog-o-sphere. And at least to me as a scientist, they communicated very clearly that they expected the reason to be the measurement error that it turned out to be. The press articles, I read about it could have been toned down a little more, but given that newspapers are competing for readers, I guess these articles were okay.

If the blogosphere starts zooming or journalists ask questions, you cannot expect scientists to keep silent. Similarly, if a scientist makes publicity, you cannot expect a journalist to ignore it. But you can expect from both groups to show restraint on studies that are not reviewed.

I often put my submitted manuscripts on my homepage, so that my colleagues can read them early and possibly give valuable feedback. (In economics it is even tradition to circulate a manuscript for over a year before submitting it to a journal. In physics the manuscript is often submitted to a public database before submitting it to a journal.) In my case, that is still pretty private, in case of Richard Muller of BEST that is equivalent to going to the press. I will not refrain from using my homepage for communication just because a blogger or journalist may read a manuscript. When I am blogging about an article, I prefer to wait until it is accepted. However, some posts contain ideas that may become part of an article later.

Let's close with the Watts et al. (2012) manuscript, this post started with. I would say that Anthony Watts naturally had the right to publish the manuscript on his blog and ask his readers for feedback. Given the large readership of WUWT this is already seeking publicity, but you cannot expect the blogger Watts not to use his blog for feedback. Immediately making a press release out of the manuscript is not especially elegant, rather one would expect a scientist to emphasis the preliminary nature of the work and state that it is not yet ready for publicity. But then, Watts is clearly not a scientist.

UPDATE I : Michael Tobis at Planet 3.0 blogged about this post and added the wise comment:

I would add: take the true skeptic’s path. The first thing to do is neither to doubt the publication nor to accept it. The first thing to do is doubt your own understanding of it.

UPDATE II: As a sequel, my latest post discusses the value of consensus in science, which is also a matter of credibility.

Further reading

• The Washington Post, Wonkblog: Two climate papers get hyped first, reviewed later. Is that a bad idea?
• Richard Black of the BBC on the same incidents
• Nature, editorial: Coping with peer rejection
• Organizers have only two options for their meetings: open or closed
• The IPCC citation policy
• Problems with and alternatives for peer review
• Blog review of Watts et al. (2012)