Tuesday 12 June 2018

Separation of review powers into feedback and importance assessment could radically improve peer review

After some blog posts about grassroots journals, it looks as if no one else will pick up the idea and I have started creating a first grassroots journal.

(It is interesting how often fear of being scooped is mentioned as reason against Open Science. Typically good ideas are not recognised before they are presented in detail and even then it takes time. At least that is my impression with the small paradigm changes I was responsible for: surrogate clouds and adaptive parameterisations.)

"It was the pits," said [economist Brian Arthur]. "Nobody there believes in increasing returns."

Susan Arthur had seen her husband returning from the academic wars before. "Well," she said, trying to find something comforting to say, "I guess it wouldn't be a revolution, would it, if everybody believed in it at the start?"

The first grassroots journal is naturally on homogenisation of climate observations. That was a good way for me to check whether the idea would work in practise. I think it does. And a concrete example is a good way for everyone to see how such a journal would work.

Using a WordPress blog works, but I am learning how to code a WordPress site to make it more user friendly and to make it easy for everyone to start such a grassroots journal, just as easy as starting a blog. (Which is really easy. Hint.)

With that it also became time to spread the idea and I have written a short guest post for the blog of the OpenUp project, an EU project on Open Publishing. After all these years of blogging that was my first guest post.

I was supposed to write about how wonderful openness is. So I wrote about how wonderful the right mix of privacy and openness are. Scientists are natural contrarians.

The main message, however, was already presented in my last blog post that if we separate the two roles of peer review — 1) feedback for the authors and 2) advising the journal whether the article is important enough for them — we will get a much more healthy quality assurance system.

In the feedback round I see no real reason to publish the content, all the (little) mistakes that were corrected. However, as it is just feedback, a friendly helping role, it would be easy publish the name of the reviewer.

In the assessment round the review itself is very interesting for others as well and is best published. Because this is judging a colleague and anonymous makes it easier to be honest. I would give the choice whether to be named to the reviewer.

Enjoy: Separation of review powers into feedback and importance assessment could radically improve peer review.