Three cheers for gatekeeping

Tuesday, 27 May 2014

Three cheers for gatekeeping

Let's try to make Bengtssongate into a learning opportunity. First he joined the Global Warming Policy Foundation, a group you should not criticise for their opinion because they officially do not have an opinion, just bias. Then he left them again writing a completely inconsistent resignation letter. In the last act of this drama a British (!) newspaper reported that a manuscript of his was once rejected. Rejected manuscripts are not news, but a fact of academic life. For more background see Climate and Stuff. If our local tabloid has such headlines; it makes me happy, nothing bad has happened in the world.

But maybe the discussion on peer review and gatekeeping is interesting. On the blog Bishop Hill, Radical Rodent (I like alliteration) summarised the main misunderstandings I would like to clarify in this post:

Personally, I am a little perplexed as to why a paper has to be peer-reviewed before it is seriously considered for reading. What is wrong with presenting your paper on completion, and let the general public decide its merits? For most papers, the readership will probably be the few who have any interest or knowledge in the subject, thus will be able to make up their own minds as to whether or not the author is publishing poppycock or not. The present “has to be peer-reviewed” ethos gets the bizarre situation where one party in an argument will not look at another’s supporting documents as they have not been peer-reviewed; a very narrow-minded attitude, I am sure you will agree.

When I was a young scientist studying clouds, I also complained about power abuse by peer reviewers. Times seem not to have changed, Sophie Lewis just blogged about similar problems. Now that I have come into contact with climate "sceptics", I have started to trust less the rationality of man and have noticed how dysfunctional discussions can be without the cultural norms of science. In short I have come to appreciate peer review a lot more.

Scientific literature

The centre of science is the scientific literature. The high quality standards (nowadays enforced by by peer review) give the scientific literature this special role. Conferences are nice to get your work known, to debate new ideas and directions and to get to know colleagues for future collaborations, but the real scientific debate is in the literature. In the best case, blogs and social media could provide a role similar to conferences while including a even broader group of people.

Getting published in the scientific literature is not so hard. Especially in the natural sciences the template is quite clear. Just do something new and interesting. Make clear that you know the scientific literature in the introduction and thus that you are sure that your work is really new. Describe the work of others (potential reviewers) fairly. Write up clearly what you did (methods) and found (results). Avoid subjective and emotional terms. Compare your results to those of others in the discussions. Avoid mixing results and discussion as well as possible. Do not draw conclusions that go beyond your findings. If the reviewers want changes, implement the good ideas, clarify the parts of the manuscript that provoked the bad suggestions, politely request that you might address extensive requests in future studies.

There are also some weird aspects. It is better to write passive sentences. While it makes the text harder to read, it suggests objectivity. It is important that the text, references and figure look neat. The reviewer cannot see your desk and computer code to judge whether you worked carefully; the neatness of the manuscripts is a proxy for this. For the same reason figures prepared in MS Excell look less professional. The points mentioned in this paragraph should, of course, be unimportant, but scientists are also humans.

Peer review

There is much wrong with peer review. Richard Horton (editor with the top medical journal the lancet) wrote (h/t esmiff):

The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.

This is all true, except for the claims of holiness, even if a little one sided. The other side is that peer review helps authors write better articles (feedback). And it helps readers find better articles (selection). And what is the alternative?

It is already possible to publish without peer review, it is called the blogosphere and some parts of it do not look pretty. It is full of suggestive and evasive language. People misrepresent or misquote what others have stated. It is so full of errors that you get the impression pseudo-sceptics do not care about the truth or their own reputation.

Unfortunately you have to read a text before you know whether it was worthwhile and sometimes you have to do research or need specialised expertise to see the problems. Hence these low quality standards, or rather no quality standards, slow down scientific progress. That is the basic reason why peer review has become a standard in science.

It hasn't always been like this (h/t Bishop Hill). In the beginning people would simply write books without any peer review. Next to the reputation of the author, the amount of work that went into writing, copying or printing and distributing the books was the main quality filter at the time. When the first scientific journals appeared the editors often made the decision by themselves and were the ones that put their reputation on the line. Only after the WWII did peer review become the standard in scientific publishing.

Without peer review every author would have to build up credibility himself. However, one can only get to know a relatively small number of people sufficiently well to know whether their work is worthwhile. Thus maybe somewhat paradoxically, peer review actually helps less known, young and new scientists to be heard. This is more important nowadays as the number of scientists is somewhat larger than in Newton's time.

As I have argued in more length before, in the same way peer review also helps fringe ideas. If you read about a weird idea at WUWT you will naturally assume it is wrong. If you read about the same weird idea in the scientific literature you will be more willing to invest your time to consider its merits.

Peer review is also important for people who are not experts in a certain field. Previously I mentioned the press as an example, but scientists from other fields also benefit by knowing that an article is probably okay. In this way peer review helps interdisciplinary research. This may be another reason why peer review has become more important in the last decades.

Concluding, peer review is gatekeeping, but its practice actually makes science more open to new people, ideas and scientists from other fields. We should not be blind to its problems and we should strive to keep on improving standards. By this I do not mean stricter standards, but that standards should be more consistently applied. Altogether, the scientific literature is clearly better with peer review as it is without.

Variable Variability

Pages

Tuesday, 27 May 2014