Open post-publication review is no substitute for pre-publication review

Thursday 26 June 2014

Open post-publication review is no substitute for pre-publication review

We have submitted a new paper. It describes how we are planning to validate the performance of homogenization methods that remove non-climatic effects from station climate data.

The way scientists write, the paper neutrally describes the new plans. For people that know the relevant scientific literature that is naturally also a critique of how we did it before, for them there is no need to spell this out and rub salt in the wounds. However, people that do not know the literature may get the impression that there are no disputes in science. Being an author on both papers and being first author of the old one, I hope I am allowed to break a little with the scientific culture and plainly describe the problems in my next post.

You could call this post-publication peer review of a scientific article. The climate dissenter may call it blog review. Post-publication review seems to be on many people's minds lately. My guess would be that this is stimulated by the increasing importance of digital publishing and social media, which make new procedures thinkable.

The most common procedure in science is that subsequent improved articles take care of problems found in published articles. It is also possible to write a so-called comment on an article, a short article that only focusses on the problems of the published paper. Being rather explicit, this is not a great way to make friends and is not used much. The authors themselves can also publish a correction or retract their articles. These procedures are all quite heavy; also these texts typically involve peer review and are printed in the journal. Because of this it is quite hard to get a comment published. So they say, I have never tried.

It may be possible to do post-publication review more loosely in the digital age. Although, while the limitation is no longer the cost of printing and shipping paper around the world, an important limitation is still the time of the reader. The problem is not getting published, but getting read (by the right people). Thus maintaining a certain quality level is still important. If it weren't we could dump the journals and all just read blogs. Does not sound like a good idea to me.

The post-publication review could be similar to the pre-publication open peer review that the European Geophysical Union (EGU) uses for some of its journals. Unfortunately, these journal do not keep the discussion open after publication. Furthermore, except for the official reviewers, the people have to sign their comments. While I understand why the editors prefer this, it reduces the number of low quality comments they have to read and moderate, I feel that also anonymous comments should be possible. Not every paper author deals with criticisms professionally.

Facilitate review after publication

Another nice example is the journal PLOS ONE. The Public Library Of Science, PLOS, is a pioneer in open-access publishing in the medical sciences. In PLOS ONE everyone can publish and the review is only for the technical correctness of the manuscript, not for its importance or impact. As far as I can judge this type of review would not be a big difference for the atmospheric sciences. I can only remember one or two manuscripts were I wrote the editor that it is a rather small incremental improvement to the literature. In almost all cases manuscripts are rejected for technical problems.

How important the expected impact of a paper is in the review may be different in other fields, economists often talk about how hard it is to get into certain journals and naturally getting published in Science or Nature is hard. In the atmospheric sciences the differences in Impact Factor between the journals are modest.

PLOS ONE performs a post-publication review by having facilities to add comments and by linking to (news) articles and blog posts that mention the PLOS ONE article. A paper that is unsurprisingly shared a lot on twitter and facebook is: Facebook Use Predicts Declines in Subjective Well-Being in Young Adults.

Review only after publication

The next level of escalation would be no peer review in advance, publish anything and only comment on the articles afterwards. This is advocated in the essay: Open Peer Review to Save the World by Philip Gibbs. I think he is serious, but this surely is an overestimation of the importance of peer review. Gibbs had manuscripts rejected because he has no academic affiliation. That is something that must not to happen. Period. Clearly there are many problems with peer review. However, this alternative model is very similar to the blogosphere and we see what kind of quality that produces.

The limitation for scientific progress is not the number of potential interesting ideas, it is the build up of reliable knowledge. Not having a peer review before publication could backfire for speciality topics and for unknown authors; such papers need the review to obtain the initial credibility to get people to take the idea seriously. You already see in the EGU open review that mostly only the assigned reviewers give their opinion and that reviews by others is rare. With the large number of scientists today, people working on projects and often changing topic and the importance of interdisciplinary research, I do not think that a return to personal credibility to judge if a paper is worth reading would be beneficial for science. Only the papers written or recommended by a hand full of well-known people would be taken seriously and the rest would struggle harder to get people to invest their time to read them.

I would argue that some selection for manuscripts and comments is important to keep quality standards. However, for authors wanting to warn their readers of shortcomings of their papers, peer review does not seem that important. I will do so in my next post.

Variable Variability

Pages

Thursday 26 June 2014