Separation of feedback, publishing and assessment of scientific studies

I once asked a friend and colleague about a wrong sentence in one of his scientific articles. He is a smart cookie and should have known better than that. His answer was that he knew it was wrong, but the peer reviewer requested that claim. The error was small and completely inconsequential for the results; no real harm was done. I wondered what I would have done.

Peer review has two roles: it provides detailed feedback on your work and it advises the editor on whether the article is good enough for the journal. This feedback normally makes the article better, but it is somewhat uncomfortable to discuss with reviewers who have a lot of power because of their second role.

Your Manuscript On Peer Review by redpen/blackpen.

My experience is that normally you can argue your case with a reviewer. Still to reach a common understanding can take an additional round of review, which means that the paper is published a few months later. In the worst case, not agreeing with a reviewer can mean that the paper is rejected and you have to submit to another journal.

It is quite common for reviewers to abuse their power by requesting their work to be cited (more). Mostly this is somewhat subtle and the citation more or less relevant. However, an anonymous reviewer once requested that I'd cite four article by one author, one of which was somewhat relevant. That does not hurt the article, but is disgusting power abuse and rewards bad behavior. My impression is that these are not all head fakes; when I write a critical review I make sure not to ask for citations to my work, but recommend some articles of colleagues instead. Multiple colleagues, not to get them into trouble.

Grassroots journals

I have started a grassroots journal on homogenization of climate data and only recently started to realize that this also produces a valuable separation of feedback, publishing and assessment of scientific studies. That by itself can lead to a much more healthy and productive quality control system.

A grassroots journal assesses published articles and manuscripts in a field of study. One could also see it as a continually up-to-date review article. At least two reviewers write a review on the strengths and weaknesses of an article, everyone can comments on parts of the article and the editors write a synthesis of the reviews. A grassroots journal does not publish the articles themselves, but collects articles published everywhere.

Every article also gets a quantitative assessment. This is similar to the current estimate of how important an article is by the journal it was able to get into. However, it does not reward people submitting the articles to a too big journal, hoping to get lucky, making unnecessary work for double reviews. For example, the publisher Frontiers reviews 2.4 million manuscripts and has to bounce about 1 million valid papers.

In case of traditional journals your manuscript only has to pass the threshold at the time of publishing. With an up-to-date rolling review of grassroots journals articles are rewarded that are of lasting value.

I would not have minded making a system without a quantitative assessment, but there are real differences between articles, the reader needs to prioritize their reading and funding agencies would likely not accept grassroots journals as replacement of the current system without it.

That is the final aim: getting rid of the current publishing system that holds science back. That grassroots journals immediately provide value is hopefully what makes the transition easier.

The more assessments made by grassroots journals are accepted the less it matters where you publish. Currently there is typically one journal, sometimes two, that have the right topic and prestige to publish in. The situation for the reader is even more terrible: you often need a specific paper and not just some paper on the topic. For this one specific paper there is one (legal) supplier. This near-monopolistic market leads to Elsevier making profits of 30 to 50% and it suppresses innovation.



Another symbol of the monopolistic market are the manuscript submission systems, which combine the worst of pre-internet paper submissions (every figure a separate file, captions in a separate file) with the internet age adage "save labor costs by letting your customers do the work" (adding the captions a second time when uploading a figure with a neat pop-up for special characters).

Separation of powers

Publishing is easy nowadays. ArXiv does this for about one dollar per manuscript. Once scientists can freely chose where to publish, the publishers will have to provide good services at reasonable costs. The most important service would be to provide a broad readership by publishing Open Access.

Maybe it will even go one step further and scientists will simply publish their manuscript on a pre-print server and tell the relevant grassroots journals where to find it. Such scientists likely still would like get some feedback from their colleagues on the manuscript. Several initiatives are currently springing up to review manuscripts before they are submitted to journals, for example, Peer Community In (PCI). Currently PCI makes several rounds until the reviewers "endorse" a manuscript so that in principle a journal could publish such a manuscript without further peer review.

With a separate independent assessment of the published article there would no longer be any need for the "feedback peer reviewers" to give their endorsement. (It doesn't hurt.) The authors would have much more freedom to decide whether the changes peer reviewers suggest are actually improvements. The authors, and not the reviewers, would decide when the manuscript is finished and can be published. If they make the wrong decisions that would naturally be reflected in the assessment. If they do not not add four citations to a peer reviewer that would not be any problem.

There is a similar initiative in the life sciences called APPRAISE, but this will only review manuscripts published on pre-print servers. Once the journals are gone, this will be the same, but I feel that grassroots journals add more immediate value by reviewing all articles on one topic. Just like a review article should review the entire literature and not a random part.

A vigorously debated topic is whether peer reviews should be open or closed. Recently ASAPbio had this discussion and comprehensively summarized the advantages and disadvantages (well worth reading). Both systems have their strengths and I do not see one of them winning.

This discussion may change when we separate feedback and assessment. Giving feedback is mostly doing the authors a favor and could more easily be done in the open. Rather than cumbersome month-long rounds of review, it would be possible to simply write an email and pick up the phone and clarify contentious points. On the other hand anonymity makes it easier to give an honest assessment and I expect this part to be mostly performed anonymously. The editors of a grassroots journal determine what is published and can thus ensure that no one abuses their anonymity.

The future

Concluding, in a decade a researcher writes an article and asks their colleagues for feedback. Once the manuscript no longer changes that much it is send to an independent proof reading service. Another firm or person takes care of the lay-out and ensures that the article can still be read in a century by making versions using open standards.

The authors decide when their manuscript is ready to be published and can be uploaded to the article repository. They send a notice to the journals that cover the topic. Journal A makes an assessment. Journals B and C copy this assessment, while journal D also uses it, but requests an additional review for a part that is important to them and they write another synthesis.

Readers add comments to the article using web annotations and the authors reply to them with clarifications. Also authors can add comments to share new insights on what was good and bad about the article.

Two years later a new study shows that one of the choices of the article was not optimal. This part was important for journal C and D and they update their assessment. The authors decide that it is relatively easy to redo their article with a better choice and that the article is sufficiently important to put in some work, they upload the updated study to the repository and the journals update their assessment.

Related reading

APPRAISE (A Post-Publication Review and Assessment In Science Experiment). A similar idea to grassroots journals, but they only want to to review pre-prints and will thus only review part of the literature. See also NPR on this initiative.

A related proposal by Gavin Schmidt: Someone C.A.R.E.S. Commentary And Replication in Earth Science (C.A.R.E.S.). Do we need a new venue for post-publication comments and replications?

Psychologist Henry L. Roediger, III on Anonymity in Scientific Publishing. A well written article that lays out all arguments, which are whether we talk about the authors, reviewers or editors. The author likes signed reviews. I feel that editors should prevent reviewers taking advantage of their anonymity.


* Photo of scientific journals by Tobias von der Haar used under a Attribution 2.0 Generic (https://creativecommons.org/licenses/by/2.0/) license.
* Graph of publishing costs by Dave Gray used under a Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) license.

Open science and science communication at #EGU18, the European Geophysical Union General Assembly

The EGU General Assembly 2018 will bring 14,500 geoscientists from all over the world to Vienna. Those are not just climate scientists plotting to take over the world. Climatology is just one of 22 disciplines present. In my last post on interesting meetings for climate data scientists, I already pointed to the relevant scientific meetings taking place for climate data scientists.

In this post I wanted to point to more general meetings (sessions, debates and short courses) that take place that may be of interest to climate scientists on Open Science, Science Communication, scientific publishing and climate change.

Such big conferences have downsides, if you meet someone you will have to immediately get contact information because chances of meeting twice by accident are small. But an advantage is the attention for topics affecting many sciences, which do not take place at more focussed workshops and smaller meetings. There are sessions on nonlinear physics, which pose methodological problems almost all geoscientists have to deal with and I attended a lot as young scientist.

There was also a time where I worked on many different topics, clouds, downscaling, homogenisation, where EGU was great for meeting all these communities in the same week. Lately I was mostly focussed on homogenisation and I have not visited EGU for some time. If you are mostly interested in one session, it is a long trip and an expensive conference for just a few talks and a poster session.

Maybe I paid less attention to this in the past, but it looks like EGU nowadays has a very wide range of meetings on Open Data, Open Code, Open Science, Publishing, Citizen Science and Science Communication. As I am thinking of destroying a multi-billion dollar scientific publishing industry, those are topics very much on my mind. So I think I will visit EGU this year and have curated a list below of meetings that I think climate scientists could be interested in. (Descriptions are often shortened.)

Bottom up scientific conferences

Old rabbits, with which the Germans do not only mean our professorial chemistry bunny, can skip to the list, but for young scientists and outsiders I thought it would be interesting to explain how such a huge conference is organised. With 14,500 geoscientists writing more than 20,000 abstracts on the work they would like to present, it is impossible for the conference organisers to determine what will happen and the content of the conference is very much a bottom-up affair.

The conference is split up in 22 disciplinary divisions and 13 divisions of general interest. One of these divisions is "Climate: Past, Present, Future". It could have been called "climate". Within these divisions you have dozens of so called "sessions", which are meetings on a specific topic.

Everyone can propose a session. For this EGU there was a call-for-sessions with a deadline in September. As far as I know the only condition is that one of the organisers of the session needs to have a PhD.

The next step is the call-for-abstracts, which for EGU2018 ends the 10th of January. Everyone can submit an abstract to a session of their liking describing what they would like to talk about. Again bottom-up.

Normally abstracts are accepted. When I was organiser of the downscaling session, I could see the number and about 1% of the abstracts was rejected. They were mostly double or empty ones, where something had gone wrong during submission. If the organiser thinks there is something wrong with your work, the abstract is normally still accepted, but you will likely get a poster.

Space is limited and the organiser can only select one third of the abstracts as talks, the others become posters. One time block with talks is one and a half hour in which six presentations can be given. The minimum size of a session is thus 18 abstracts. If your session gets less, the divisions leaders will merge your session with another one on a similar topic. Getting to 18 abstracts is the main barrier to organising your own session.

Talks are best for broadcasting a new result, you reach more people, but there is only time for a few questions and thus little feedback. Posters are much better for feedback. As a convener an important criterion for making an abstract a talk or a poster is thus the stage the study seemed to be in. In the early phase feedback is important, if the work is finished broadcasting is important. In addition talks are typically for studies of more general interest and if it is known how well someone talks that is also an important consideration. Thus if you want to get talk, make sure to mention some results to make clear you are in the final stages and make sure the abstract is clear and contains no typos, which are proxies for being able to present your work clearly.

The posters are EGU are typically well visited, especially the main evening poster session with free beer and wine to get people talking. Personally I spend most of my time at the posters. If a talk is not interesting, 15 precious minutes are gone, if a poster is not interesting you just walk along.

Some sessions at EGU have a system been a talk and a poster called a PICO session. Here people present their work in a 2-minute talk and afterwards every presenter stands next to a touch screen with the presentation for detailed discussions. The advantage of a poster over a PICO is that the poster is up all day.



Next to these sessions where people present their latest, you can also reserve rooms for splinter meetings to talk with each other or organise short courses. Many of the interesting meetings listed below are short courses.

Science

Great Debate 4 Low-risk geo-engineering: are techniques available now?

With the Paris agreement, a majority of the world’s countries have agreed to keep anthropogenic warming below 2 °C. According to the Intergovernmental Panel on Climate Change^*, this target would require not only reducing all man-made greenhouse gas emissions to zero but also removal of large amounts of carbon dioxide from the atmosphere, or other type of geo-engineering techniques. The issue of geo-engineering has been heavily debated during the last years and we are therefore asking: Are the potential risks with geo-engineering sufficiently known? Are safe geo-engineering techniques available? Are they available now?

This debate will address these questions of crucial importance for today’s society. It will discuss the most recent discoveries of geo-engineering techniques, their potential to reduce global warming and their potential risks.

^* I am not sure whether the IPCC states this. The scenarios that stay below 2 °C do have carbon dioxide removal. But scenarios are just that, scenarios, not predictions for the future.

I think we urgently need to talk about a geo-intervention. There is no reason to wait until the Earth has warmed 2 °C, climate change does unacceptable damage right now.

Great Debate 2 Hands on or hands off?

A great debate on whether scientists should get involved in policy.

In recent years there has been a growing distrust of experts in the public imagination which has been expressed in numerous debates from Brexit to the US presidential election. This gives rise to serious questions about the role of scientists in policy making and the political sphere. As geoscientists, our disciplines can have a real impact on the way humanity organises itself, so what should our role in that be? There are serious tensions here between the desire for our knowledge to have real impact and make a difference, the need for scientific detachment and objectivity, and respect for broader perspectives and for democracy itself.

The key questions for this debate are:

• Should geoscientists restrict themselves to knowledge generation and stay out of the policy world?
• Or should we be getting involved and making change happen?
• Should our voices as experts be heard louder than others?
• Or does evidence-based policy undermine democracy?
• Should we be hands on or keep our hands off

Conferences are busy, so let me answer the questions so you do not have to go.

Four of the five organisers are from the UK, but I hope that at least outside of Anglo-America it is uncontroversial for scientists to inform the public and policy makers of their findings. Scientists are humans and have human rights, including free speech. Germany and several other European countries have even set up climate service centres to facilitate the flow of information from science to groups that need to adapt to climatic changes.

When it goes further, trying to convince people of certain solutions, please let go of your saviour complex, you will mostly like not achieve much. The way scientists are trained to think and communicate works well for science, but it is not particularly convincing outside of it. The chance you are good at convincing people is not much better than the chance of some random dude or grandma down the road.

When it comes to informing people of our findings our voice should naturally be louder than that of groups misinforming people. In countries with a functioning media that does not need to be particularly loud. The opposite of evidence-based policy is misinformation-based policy. It is clearly less democratic and an abuse of power to set up a misinformation campaign to get your way politically because the public would not support your policies if they knew the truth. That is a violation of the self-determination of people to control their lives.

Educational and Outreach Symposia Session Geoethics: ethical, social and cultural implications of geoscience knowledge, education, communication, research and practice

 

Educational and Outreach Symposia Session Vision for Earth Observations in 2040

Educational and Outreach Symposia is a surprising place for this session. I hope the right people find it.

As both Earth science and technology advance while the expectations for the extent, quality, and timeliness of environmental information to be provided to the world’s population increases, the opportunity exists to harness the increased knowledge and capability to improve those products and services.

The World Meteorological Organization has made important contributions in making the connection between knowledge and products in the areas of weather, water, and climate through its periodic visions, most recently the Vision for the Global Observing System (GOS) in 2025. The WMO is now in the process of doing an update for the 2040 time frame, taking into account both surface and space-based measurements.

In this session, presentations that look ahead to the 2040 time frame and address expected observational capability that can realistically be expected to be available in that time frame, the expected demand for products and services informed by observational data that may be required for public use in that time frame, and mechanisms for connecting the two are all sought. Presentations for this session can address the full range of products for Earth System Science and are not limited to those addressed by the WMO in its development of the 2040 vision.

I hope a global climate station reference network will be part of this vision for 2040.

Interdisciplinary Session Big data and machine learning in geosciences

This session aims to bring together researchers working with big data sets generated from monitoring networks, extensive observational campaigns and extremely detailed modeling efforts across various fields of geosciences. Topics of this session will include the identification and handling of specific problems arising from the need to analyze such large-scale data sets, together with and methodological approaches towards automatically inferring relevant patterns in time and space aided by computer science-inspired techniques. Among others, this session shall address approaches from the following fields:

• Dimensionality and complexity of big data sets
• Data mining and machine learning
• Deep learning in geo- and environmental sciences
• Visualization and visual analytics of big data
• Complex networks and graph analysis
• Informatics and data science

Interdisciplinary PICO Session R’s deliberate role in Earth sciences

 

Interdisciplinary Session Citizen Science in the Era of Big Data

I wish they were a bit clearer on what kind of statistics specially for Big Data they are thinking of. I guess with lots of data it is easy to a result that is statistically significant, but physically negligibly small and not interesting. If you start analysing the data like a fishing expedition you should be extra careful not to fall into the multiple testing gap. It may be that they mean that with "Big Data approaches."

Citizen Science (the involvement of laypeople in scientific processes) is gaining momentum in one discipline after another, thereby more and more data on biodiversity, earthquakes, weather, climate, health issues among others are being collected at different scales. In many cases these datasets contain huge amounts of data points collected by various stakeholders. There is definitely power in numbers of data points, however, the full potential of these datasets is not realized yet. Traditional statistics often fail to utilize these prospects. Statistics for Big Data can unveil patterns hidden that are otherwise would not be visible in datasets. Since Big Data approaches and citizen science are still developing fields, most projects miss Big Data analyses.

In this session we are looking for successful approaches of working with Big Data in all fields of citizen science. We want to ask and find answers to the following questions:

• Which Big Data approaches can be used in citizen science?
• What are the biggest challenges and how to overcome them?
• How to ensure data quality?
• How to involve citizen scientists in Big Data Analyses, or is it possible?

Scientific publishing

Great Debate 1 Who pays for Scientific Publishing?

This Great Debate will address the following questions: whether the profits generated by traditional publishers are justifiable and sustainable, to what extent scientists should contribute to the business, what are the current and future alternatives, and what role will preprint servers play?

See also my recent post on the new preprint servers for the Earth Sciences and the townhall meeting below on self-archiving and EarthArXiv.

Townhall Meeting EarthArXiv - a preprint server for the Earth Sciences

Preprints and preprint servers are set to revolutionise and disrupt the standard approaches to scholarly publishing in the Earth Sciences. Yet, despite being widely-used and demonstrably successful in several other core science disciplines, the concept of preprints is new to many Earth Sciences. As a result, education is needed, such that Earth Scientists can benefit from the use of preprints and preprint servers. In this townhall we will introduce the general concepts of preprints and preprint servers, illustrating this with a demonstration of EarthArXiv, a community-led preprint server. We will also lead a general discussion of the use of preprints.

PICO Session Future of (hydrological) publishing

This session was one reason to write this blog post. It sounds really interesting and it is somewhat hidden by being in the Hydrology Division, where non-hydrologists may miss it. This could be a good place for my coming out with the idea of grassroots scientific publishing, where the scientific community takes back control of the quality assessment, beginning with making more informative open reviews of already published articles.

In recent years, the current and future system of scientific publishing has been heavily debated. Most of these discussions focused on criticizing aspects of the current system such as:

• the scientific publishing industry being one of the most profitable branches (Guardian, 2017) in media, because the scientific community basically does all the work for free
• the peer review system being corrupted, or at least not functioning perfectly
• the limited access to scientific papers due to its current business model
• the surging number of submitted papers in recent years, especially with strict publication requirements for PhD candidates. This is putting more pressure on editors, reviewers and readership, and will decrease the visibility and impact of each publication.

Times are changing, which can be seen in the increased demand and supply for open access publishing. However, we believe there might be plenty of other ideas and suggestions on how to improve scientific publishing. We invite and challenge everyone from the scientific community to propose ideas on how to do so in 5 minute presentations. Afterwards we will continue the discussion to answer questions such as: Who needs to pay for reading our work? Who should publish our work? How to cope with the excessive amount of submitted papers? Should we even be publishing?

Short course What are the key problems in Climate Science?

Climate science is a wide discipline that encompasses many of the EGU divisions, yet it is not always easy to know what the key problems are outside of your own specific area. ... During the short course, four climate experts from different divisions will introduce the “key problems” in their discipline, giving you an overview of what the current “hot topics” are. This course will provide you with enough background to venture into other divisions during the rest of the meeting. The floor will then be open for questions and discussion with our experts.

With so many disciplines together EGU would theoretically be an important place to learn about problem in other fields and see how that fits to yours.

However, the talks at EGU are very short, just 12 minutes. They do not leave much time for an introduction and are thus hard to follow for outsiders. It may be nice to extend the idea of this short course with just four hot topics to many more topics. Make it into a science slam, where you do not talk about your own work, but introduce the field in a way an outsider can get it.

It looks like these four key problems and their speakers are selected by the conveners. A science slam could be open to all like the normal talks.

Open Science

Townhall Meeting OSGeo Townhall: Open Science demystified

OSGeo is hosting this Townhall event to support the collaborative development of open source geospatial software by promoting sustainable Open Science within EGU. The Open Source Geospatial Foundation, or OSGeo, is a not-for-profit umbrella organization for Free and Open Source geospatial tools, including QGIS, gvSIG, GRASS GIS, Geoserver and many others.

The paradigm of Open Science is based on the tiers Open Access, Open Data and Free Open Source Software (FOSS). However, the interconnections between the tiers remain to be improved. This is a critical factor to enable Open Science.

This Townhall meeting reaches out all across EGU, especially welcoming Early Career Scientists, to network and discuss the current challenges and opportunities of the FOSS tier, including:

• the easy approach to choosing software licences
• recognition for scientific software: how to write a software paper?
• software life cycle: who will maintain your software after you've finished your PhD and found a decent job?
• funding software development: evolving software begun for your own research needs into something larger, that serves other's needs, and boosts your scientific reputation.
• software reviews: how to set up software development such that other developers get involved in an early stage?
• how can OSGeo help you with all these questions?

Short course How to find and share data in geosciences?

This short course aims to present some tips and tricks to accelerate the process of finding, processing and sharing the Geosciences data. We will also discuss the importance of open science and the opportunities it provides.

Short course Writing reproducible geoscience papers using R Markdown, Docker, and GitLab

I think code reproducibility is overrated. It is much stronger to make an independent reproduction and if the result depends on minute details being the same, it is mostly likely not a useful result. But the most of the same methods can be useful to speed up scientific progress. Sharing data and code is wonderful and helps other scientists get going faster. The code should thus preferably also run somewhere else.

Reproducibility is unquestionably at the heart of science. Scientists face numerous challenges in this context, not least the lack of concepts, tools, and workflows for reproducible research in Today's curricula.

This short course introduces established and powerful tools that enable reproducibility of computational geoscientific research, statistical analyses, and visualisation of results using R (http://www.r-project.org/) in two lessons:

1. Reproducible Research with R Markdown
Open Data, Open Source, Open Reviews and Open Science are important aspects of science today. In the first lesson, basic motivations and concepts for reproducible research touching on these topics are briefly introduced. During a hands-on session the course participants write R Markdown documents, which include text and code and can be compiled to static documents (e.g. HTML, PDF).

R Markdown is equally well suited for day-to-day digital notebooks as it is for scientific publications when using publisher templates.

To understand the rest of the description, I need to explain what Docker means:

Docker is a tool that can package an application and its dependencies in a virtual container that can run on any Linux server. This helps enable flexibility and portability on where the application can run, whether on premises, public cloud, private cloud, bare metal, etc.

Gitlab is a collaborative coding system based on the versioning system [[Git]]. Comparable to be probably better known [[GitHub]] and [[Bitbucket]].

2. GitLab and Docker
In the second lesson, the R Markdown files are published and enriched on an online collaboration platform. Participants learn how to save and version documents using GitLab (http://gitlab.com/) and compile them using [[Docker]] containers (https://docker.com/). These containers capture the full computational environment and can be transported, executed, examined, shared and archived. Furthermore, GitLab's collaboration features are explored as an environment for Open Science.

P.S. Those homepages really suck big time, except if their goal is to scare away anyone who is a hard core coder and already knows the product. That is why I mostly linked to Wikipedia.

Short course Building and maintaining R packages

R is a free and open software that gained paramount relevance in data science, including fields of Earth sciences such as climatology, hydrology, geomorphology and remote sensing. R heavily relies on thousands of user-contributed collections of functions tailored to specific problems, called packages. Such packages are self-consistent, platform independent sets of documented functions, along with their documentations, examples and extensive tutorials/vignettes, which form the backbone of quantitative research across disciplines.

This short course focuses on consolidated R users that have already written their functions and wish to i) start appropriately organizing these in packages and ii) keep track of the evolution of the changes the package experiences. While there are already plenty of introductory courses to R we identified a considerable gap in the next evolutionary step: writing and maintaining packages.

Short course Improving statistical evaluations in the geosciences

I love the (long) description of topics. Looks like just what a geo-scientist needs.

During my studies I got lucky. Studying physics, the only statistics we got was some error propagation for lab work. Somehow I was not happy with that and I found a statistics course in the sociology department. There was not much mathematics, one student even asked what the dot between X and Y was. I did not even understand the question, but the teacher casually answered that that was the multiplication sign. Maybe out of need it focussed on the big ideas, on the main problems and typical mistakes people make. It looks like this could be a similar course, but likely with more math.

Session Open Data, Reproducible Research, and Open Science

Open Data and Open Science not only address publications, but scientific research results in general, including figures, data, models, algorithms, software, tools, notebooks, laboratory designs, recipes, samples and much more.

Furthermore, they relate to the communication, review, and discussion of research results and consider changing needs regarding incentives, quality assessment, metrics, impact, reputation, grants and funding. Thus Open Data and Open Science encompass licensing, policy-making, infrastructures and scientific heritage, while safeguarding the dynamic nature of science and its evolving forms.
...
The speakers present success stories, failures, best practices, solutions and introduce networks. It is aimed to show how researchers, citizens, funding agencies, governments and other stakeholders can benefit from Open Data, Reproducible Research, and Open Science in various flavors, acknowledging the drawbacks and highlighting the opportunities available for geoscientists.

The session shall open a space to exchange experiences and to present either successful examples or failed efforts. Learning from others and understanding what to adopt and what to change are to help towards own undertakings and new initiatives, so that they become successes.

Educational and Outreach Symposia Session Promoting and supporting equality of opportunities in geosciences

Following the success of previous years, this session will be exploring reasons for the existence of underrepresentation of different groups (cultural, national and gender) by welcoming a debate with scientists, decision-makers and policy analysts related to geosciences.

The session will be focusing on both remaining obstacles that contribute to underrepresentation and on best practices and innovative ideas to tackle obstacles.

Science communication

Short course Help! I'm presenting at a scientific conference!

Sounds like this short course on giving a scientific presentation is tailored to newbies, although the seniors could also use some help. The seniors are hardest to change, they have learned that they have a highly motivated captive audience an that after a crappy talk everyone will pretend it was a good one.

Presenting at a scientific conference can be daunting for early career scientist and established. How can you optimally take advantage of those 12 minutes to communicate your research effectively? How do you cope with nervousness? What happens if someone asks a question that you don’t think you can answer? Is your talk tailored to the audience?

Giving a scientific talk is a really effective way to communicate your research to the wider community and it is something anyone can learn to do well! This short course provides the audience with hands-on tips and tricks in order to make your talk memorable and enjoyable for both speaker and audience.

Short course Once upon a time in Vienna

A short course on story telling, which is really important for readable prose. Although this blog post is probably not the best place to make this case.

This is an interactive workshop led by a professional communications facilitator and writer, and academics with a range of earth science outreach experience. Through a combination of expert talks, informal discussion, and practical activities, the session will guide you through the importance of storytelling, how to find exciting stories within your own research, and the tools to build a memorable narrative arc.

Short course Rhyme your research

After seeing the term "experienced science-poet" I was forced to include this short course. They missed the opportunity to write the description as poem.

Poetry is one of the oldest forms of art, potentially even predating literacy. However, what on Earth does it have to do with science? One is usually subjective and emotive, whilst the other (for the most part) is objective and empirical. However, poetry can be a very effective tool in communicating science to a broader audience, and can even help to enhance the long-term retention of scientific content. During this session, we will discuss how poetry can be used to make (your) science more accessible to the world, including to your students, your professors, your (grand)parents, and the general public.

Writing a poem is not a particularly difficult task, but writing a good poem requires both dedication and technique; anyone can write poetry, but it takes practice and process to make it effective. In this session, experienced science-poets will discuss the basics of poetry, before encouraging all participants to grab a pen and start writing themselves. We aim to maximise empowerment and minimise intimidation. Participants will have the opportunity to work on poems that help to communicate their research, and will be provided with feedback and advice on how to make them more effective, engaging and empathetic. Those who wish to do so may also recite their creations during the “EGU Science Poetry Slam 2018”.

Educational and Outreach Symposia Session Scientists, artists and the Earth: co-operating for a better planet sustainability

As communicators and artists, ​we have a shared responsibility ​to raise awareness of the importance of planet sustainability. ​ Educating people ​in this regard has normally been executed through traditional educational method​s.​ ​But ​there is evidence that science-art collaborations play a vital role in contributing to this issue, through the emotional and human connection that the arts can provide. This session,​ already in​ its fourth edition, has presented interesting ​and progressive ​​art science collaborations across a number of disciplines focussed on representing Earth science content. ​We have witnessed that climate change, natural hazard, meteo​rol​o​gy, palaeontology, earthquakes and volcanoes, geology have ​been successfully presented through music, visual art, photography, theatre, literature, digital art, ​where the artists ​explored new ​practices and methods in their work with scientists. ​A fundamental part of all art is the presentation of their final work. Then we provide a related 'performative session', to allow artists perform excerpts of their work and fully reveal the impact of this work in communicating the bigger planet sustainability message. This related session is entitled “A pilot-platform for performing your Earth&Art work”​​.

Short course Visualization in Earth Science: best practices

This short course is co-organized by the ESSI division: Earth & Space Science Informatics.

With constantly growing data sizes, both an effective visualization, as well as an efficient data analysis are getting more and more important. Different tasks in visualization require different visualization strategies. Geoscience data presents particular challenges, being typically large, multivariate, multidimensional, time-varying and uncertain. This short course aims at the presentation and demonstration of commonly available visualization tools, that are especially well suited to analyze earth science data sets. We at DKRZ -- the German Climate Computing Centre -- have many years of experience in the visualization of earth science data sets, and the goal of this workshop is to pass this knowledge on to you. We will show, explain and demonstrate the tools live, with which we work in our daily routine, and show you how to create effective and meaningful visualizations using free software.

Short course How to cartoon science

 

 Short course Science for Policy: What is it and how can scientists become involved in policy processes?

Organised by the EGU policy expert Chloe Hill.

Part 1: will focus on basic science for policy and communication techniques that can be used to engage policymakers. It will be of particular interest to anyone who wants to make their research more policy relevant and learn more about science-policy.

Part 2: will include invited speakers who will outline specific EU processes and initiatives and explain how scientists can become involved with them.

Short course Communicating your research to teachers, schools and the public - interactively

If you are serious about communicating your research to teachers, schools and the public then you should know something about these audiences, be familiar with the most effective ways of engaging with each of them and be clear about what the ‘take away’ messages would be. ... Methods of engaging the public and families through open days and similar events are different again, and usually use a range of activities to engage and educate at the same time. We will discuss insights and strategies for these different audiences and ask you to have a go yourself.

Short course Debunking myths and fake news: how can geoscientists fight misinformation and false claims

Maybe you’ve had an argument on social media with a climate change denier who is convinced the Earth is not warming. Or maybe you’ve received an email from a scared relative forwarding you a piece from an unreliable website about how total solar eclipses produce harmful rays that can make you blind. How do you go about convincing them they are mistaken without them holding on even more to their false beliefs? In the age of Brexit and Trump, of fake news and of expert snubbing, geoscientists have a role to play in tactfully fighting misinformation related to the Earth, space and planetary sciences. This short course will explore ways in which researchers can promote evidence and facts, prevent fake news from spreading, and successfully debunk false claims.

Short course Connect2Communicate: communicating your message with charisma, clarity and conviction

Making use of established techniques from the world of theatre and improvisation, this session will enable participants to make genuine connection with their audience.

Short course Science writing: selling your research through press releases and articles

Our press office once organised a short workshop on writing press releases, which was given by a former journalist. He could explain well what a journalist wanted from a press release, but did not understand that the interests of scientists are different. This course may be better, it is given by scientists.

The course will consist of: an introduction on how to identify a good science story; general tips on how to write with clarity and flair; an introduction on how to go about promoting your work via press releases and working with embargoes; tips on working with press officers and journalists; practical exercises on headline writing; and practical exercises about turning abstracts into press releases.

Short course Communicating geoscience to the media

The news media is a powerful tool to help scientists communicate their research to wider audiences. However, at times, messages in news reports do not properly reflect the real scientific facts and discoveries, resulting in misleading coverage and wary scientists. This is especially problematic in fields such as climate science, where climate skeptics can twist the research results to draw conclusions that are baseless. A way scientists have to prevent misleading or even inaccurate coverage is to improve the way they communicate and work with journalists. In this short course, co-organised with the CL and CR divisions, we will bring together science journalists and researchers with experience working with the media to provide tips and tricks on how scientists can better prepare for interviews with reporters. We will also provide pointers on how to ensure a smooth working relationship between researchers and journalists by addressing the needs and expectations of both parties. The focus will be on climate topics, but much of the advice would be applicable to other geoscience areas.

Educational and Outreach Symposia Session ECORD IODP Outreach: Past, Present and Future

The International Ocean Discovery Program is an international programme that works to explore the oceans and the rocks beneath them. ...

This session addresses the formats by which we disseminate scientific information and discoveries arising from ocean drilling – what have we done in the past, what are we doing now, and what ideas do we have for the future engagement of students with ocean research drilling. Experiences and examples of best practice illustrated in poster or oral format will present school teachers, university lecturers and researchers that describe their outreach efforts in the lab, field and geoscience classrooms to promote high-quality geoscience education at all levels.

Educational and Outreach Symposia Session Games for Geoscience

Games have the power to ignite imaginations and place you in someone else’s shoes or situation, often forcing you into making decisions from perspectives other than your own. This makes them potentially powerful tools for communication, through use in outreach, disseminating research, in education at all levels, and as a method to train the public, practitioners and decision makers in order to build environmental resilience. The session is a chance to share your experiences and best practice with using games to communicate geosciences, be they analogue, digital and/or serious games.

Educational and Outreach Symposia Session Communication and Education in Geoscience: Practice, Research and Reflection

Do you consider yourself a science communicator? Does your research group or institution participate in public engagement activities? Have you ever evaluated or published your education and outreach efforts?

Scientists communicate to non-peer audiences through numerous pathways including websites, blogs, public lectures, media interviews, and educational collaborations. A considerable amount of time and money is invested in this public engagement and these efforts are to a large extent responsible for the public perception of science. However, few incentives exist for researchers to optimize their communication practices to ensure effective outreach. This session encourages critical reflection on science communication practices and provides an opportunity for science communicators to share best practice and experiences with evaluation and research in this field.

Related reading

Slides of the talk: How to convene a session at the EGU General Assembly by Stephanie Zihms, Roelof Rietbroek and Helen Glaves.

EGU2018 and its call-for-abstracts.

The call-for-session of EMS2018 is currently open. Suggestions for improvements of the description of the "Climate monitoring: data rescue, management, quality and homogenization" session are welcome.

The fight for the future of science in Berlin. My report of thisyear's conference on scholarly communication, which lots of ideas and initiatives on Open Science and publishing.

Where is a climate data scientist to go in 2018?

Publish or perish is illegal in Germany, for good reason

Had Albert Einstein died just after his wonder year 1905 he would only have had a few publications on special relativity, the equivalence of mass and energy, Brownian motion and Photoelectric Effect on his name and would nowadays be seen as a mediocre researcher. He got the Nobel prize in 1921 "for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect", not for relativity, not for Brownian motion. This illustrates how hard it is to judge scientific work, even more than a decade afterwards, much less in advance.

Managing scientists is hard. It is nearly impossible to determine who will do a good job, who is doing a good job and even whether someone did a good job in the past. The last decades science managers have largely given up trying to assess how good a scientist is in most of the world and instead assess how many articles they write and how high the prestige is of the journals the articles appear in.

Unsurprisingly, this has succeeded in increasing the number of articles scientists write. Especially in America scientists are acutely aware that they have to publish or perish.

Did this hurt scientific progress? It is unfortunately impossible to say how fast science is progressing and how fast it could progress. The work is about the stuff we do not understand yet after all. The big steps, evolution, electromagnetism, quantum mechanics, have become rare the last decades. Maybe the low hanging fruit is simply gone. Maybe it is also modern publish-or-perish management.

There are good reasons to expect publish-or-perish management to be detrimental.
1. The most basic reason: The time spend writing and reading articles the ever increasing number of articles is not spend on doing research. (I hope no one is so naive as to think that the average scientist actually became several times more productive.)
2. Topics that quickly and predictably lead to publications are not the same topics that will bring science forward. I personally try to write a mix because only working on more risky science you expect is important is unfortunately too dangerous.
3. The stick and carrot type of management works for manual labor, but for creative open-ended work it is often found to be detrimental. For creative work mastery and purpose are the incentives.

German science has another tradition, trusting scientists more and focusing on quality. This is expressed in the safeguards for good scientific practice of the German Science Foundation (DFG). It explicitly forbids the use of quantitative assessments of articles.

Universities and research institutes shall always give originality and quality precedence before quantity in their criteria for performance evaluation. This applies to academic degrees, to career advancement, appointments and the allocation of resources. …

criteria that primarily measure quantity create incentives for mass production and are therefore likely to be inimical [harmful] to high quality science and scholarship. …

Quantitative criteria today are common in judging academic achievement at all levels. … This practice needs revision with the aim of returning to qualitative criteria. … For applications for academic appointments, a maximum number of publications should regularly be requested for the evaluation of scientific merit.

For a project proposal to the German Science Foundation this "maximum number" means that you are not allowed to list all your publications, but only the 6 best ones (for a typical project, smaller projects even less).

[UPDATE. This limit has unfortunately now been increased to 10. They say the biologists are to blame.]

While reading the next paragraphs, please hear me screaming YES, YES, YES in your ear at an unbearable volume.

An adequate evaluation of the achievements of an individual or a small group, however, always requires qualitative criteria in the narrow sense: their publications must be read and critically compared to the relevant state of the art and to the contributions of other individuals and working groups.

This confrontation with the content of the science, which demands time and care, is the essential core of peer review for which there is no alternative. The superficial use of quantitative indicators will only serve to devalue or to obfuscate the peer review process.

I fully realize that actually reading someone’s publications is much more work than counting them and that top scientists spend a large part of their time reviewing. In my view that is a reason to reduce the number of reviews and trust scientists more. Hire people who have a burning desire to understand the world, so that you can trust them.

Sometimes this desire goes away when people get older. For the outside world this is most visible in some older participants of the climate “debate” who hardly produce new work trying to understand climate change, but use their technical skills and time to deceive the public. The most extreme example I know is a professor who was painting all day long, while his students gave his lectures. We should be able to get rid of such people, but there is no need for frequent assessments of people doing their job well.

You also see this German tradition in the research institutes of the Max Planck Society. The directors of these institutes are the best scientists of the world and they can do whatever they think will bring their science forward. Max Planck Director Bjorn Stevens describes this system in the fourth and best episode of the podcast Forecast. The part on his freedom and the importance of trust starts at minute 27, but best listen to the whole inspiring podcast about which I could easily write several blog posts.

Stevens started his scientific career in the USA, but talks about the German science tradition when he says:

I can think of no bigger waste of time than reviewing Chris Bretherton’s proposals. I mean, why would you want to do that? They guy has shown himself to have good idea, after good idea, after good idea. At some point you say: go doc, go! Here is your budget and let him go. This whole industry that develops to keep someone like Chris Bretherton on a leash makes no sense to me.

Compare scientists who sets priorities within their own budgets with scientists who submit research proposals judged by others. If you have your own budget you will only support what you think is really important; if you do A, you cannot do B. Many project proposals are written to fit into a research program, because a colleague wants to collaborate and apart from the time wasted on writing it, there are no downsides for asking for more funding. If you have your own budget, the person with the most expertise and with the most skin in the game decides. This while they call the project funding, where the deciders have no skin in the game, competitive. It is Soviet style planning; that it works at all shows the dedication and altruism of the scientists involved. Those are scientists you could simply trust.

I hope this post will inspire the scientific community to move towards more trust in scientists, increase the fraction of unleashed researchers and reduce the misdirected quantitative micro-management. Please find below the full text of the safeguards of the German Science Foundation on performance evaluation; above I had to skip many worthwhile parts.

Tweet

---

Recommendation 6: Performance Evaluation

Universities and research institutes shall always give originality and quality precedence before quantity in their criteria for performance evaluation. This applies to academic degrees, to career advancement, appointments and the allocation of resources.

Commentary
For the individual scientist and scholar, the conditions of his or her work and its evaluation may facilitate or hinder observing good scientific practice. Conditions that favour dishonest conduct should be changed. For example, criteria that primarily measure quantity create incentives for mass production and are therefore likely to be inimical to high quality science and scholarship.

Quantitative criteria today are common in judging academic achievement at all levels. They usually serve as an informal or implicit standard, although cases of formal requirements of this type have also been reported They apply in many different contexts: length of Bachelor, Master or PhD thesis, number of publications for the Habilitation (formal qualification for university professorships in German speaking countries), as criteria for career advancements, appointments, peer review of grant proposals, etc. This practice needs revision with the aim of returning to qualitative criteria. The revision should begin at the first degree level and include all stages of academic qualification. For applications for academic appointments, a maximum number of publications should regularly be requested for the evaluation of scientific merit.

Since publications are the most important “product” of research, it may have seemed logical, when comparing achievement, to measure productivity as the number of products, i.e. publications, per length of time. But this has led to abuses like the so-called salami publications, repeated publication of the same findings, and observance of the principle of the LPU (least publishable unit).

Moreover, since productivity measures yield little useful information unless refined by quality measures, the length of publication lists was soon complemented by additional criteria like the reputation of the journals in which publications appeared, quantified as their “impact factor” (see section 2 5).

However, clearly neither counting publications nor computing their cumulative impact factors are by themselves adequate forms of performance evaluation. On the contrary, they are far removed from the features that constitute the quality element of scientific achievement: its originality, its “level of innovation”, its contribution to the advancement of knowledge. Through the growing frequency of their use, they rather run the danger of becoming surrogates for quality judgements instead of helpful indicators.

Quantitative performance indicators have their use in comparing collective activity and output at a high level of aggregation (faculties, institutes, entire countries) in an overview, or for giving a salient impression of developments over time. For such purposes, bibliometry today supplies a variety of instruments. However, they require specific expertise in their application.

An adequate evaluation of the achievements of an individual or a small group, however, always requires qualitative criteria in the narrow sense: their publications must be read and critically compared to the relevant state of the art and to the contributions of other individuals and working groups.

This confrontation with the content of the science, which demands time and care, is the essential core of peer review for which there is no alternative. The superficial use of quantitative indicators will only serve to devalue or to obfuscate the peer review process.

The rules that follow from this for the practice of scientific work and for the supervision of young scientists and scholars are clear. They apply conversely to peer review and performance evaluation:

• Even in fields where intensive competition requires rapid publication of findings, quality of work and of publications must be the primary consideration. Findings, wherever factually possible, must be controlled and replicated before being submitted for publication.
• Wherever achievement has to be evaluated — in reviewing grant proposals, in personnel management, in comparing applications for appointments — the evaluators and reviewers must be encouraged to make explicit judgements of quality before all else. They should therefore receive the smallest reasonable number of publications — selected by their authors as the best examples of their work according to the criteria by which they are to be evaluated.

---

Related information

Nature on new evaluation systems in The Netherlands and Ireland: Fewer numbers, better science

Episode 4 of Forecast with Max Planck Director Bjorn Stevens on clouds, aerosols, science and science management. Highly recommended.

Memorandum of the German Science Foundation: Safeguarding Good Scientific Practice. English part starts at page 61.

On of my first posts explaining why stick and carrot management makes productivity worse for cognitive tasks: Good ideas, motivation and economics

* Photo of Albert Einstein at the top is in the public domain.

Grassroots scientific publishing

These were the weeks of peer review. Sophie Lewis wrote her farewell to peer reviewing. Climate Feedback is making it easy for scientists to review journalistic articles with nifty new annotation technology. And Carbon Brief showed that while there is a grey area, it is pretty easy to distinguish between science and nonsense in the climate "debate", which is one of the functions of peer review. And John Christy and Richard McNider managed to get an article published, which I would have advised to reject as reviewer. A little longer ago we had the open review of the Hansen sea level rise paper, where the publicity circus resulted in a-scientific elements spraying their graffiti on the journal wall.

Sophie Lewis writes about two recent reviews she was asked to make. One where the reviewers were negative, but the article was published anyway by the volunteer editor and one case where the reviewers were quite positive, but the manuscript was rejected by a salaried editor.

I have had similar experiences. As reviewer you invest your time and heart in a manuscript and root for the ones you like to make it in print. Making the final decision naturally is the task of the editor, but it is very annoying as a reviewer to have the feeling your review is ignored. There are many interesting things you could have done in that time. At least nowadays you get to see the other reviews and hear the final decision more often, which is motivating.

The European Geophysical Union has a range of journals with open review, where you can see the first round of reviews and anyone can contribute reviews. This kind of open review could benefit from the annotation system used by Climate Feedback to review journalistic articles; it makes reviewing easier and the reader can immediately see the text the review refers to. The open annotation system allows you to add comments to any webpage or PDF article or manuscript. You can see it as an extra layer on top of the web.

The reviewer can select a part of the text and add comments, including figures and links to references. Here is an annotated article in the New York Times that Climate Feedback found to be scientifically very credible, where you can see the annotate system in action. You can click on the text with a yellow background to see the corresponding comment or click on the small symbol at the top right to see all comments. (Examples of articles with low scientific credibility are somehow mostly pay-walled; one would think that the dark money behind these articles would want them to be read widely.)

I got to know annotation via Climate Feedback. We use the annotation system of Hypothes.is and this system was actually not developed to annotate journalistic articles, but for reviewing scientific articles.

The annotation system makes writing a review easier for the reviewer and makes it easier to read reviews. The difference between writing some notes on an article for yourself and a peer review becomes gradual this way. It cannot take away having to read the manuscript and trying to understand it. That takes most time, but this is the fun part, reducing time time for the tedious part makes it more attractive to review.

Publishing and peer review

Is there a better way to review and publish? The difficult part is no longer the publishing. The central part that remains is the trust of a reader in a source.

It starts to become ironic that the owners of the scientific journals are called "scientific publishers" because the main task of a publisher is nowadays no longer the publishing. Everyone can do that nowadays with a (free) word processor and a (free) web page. The publishers and their journals are mostly brands nowadays. The scientific publisher, the journal is a trusted name. Trust is slow to build up (and easy to lose), producing huge barriers to entry and leading to near monopoly profits of scientific publishing houses of 30 to 40%. That is tax-payer money that is not spend on science and promotes organization that prefer to keep science unused behind pay-walls.

Peer review performs various functions. It helps to give a manuscript the initial credibility that makes people trust it, that makes people willing to invest time in it to study its ideas. If the scientific literature would be as abominable as the mitigation skeptical blog Watts Up With That (WUWT) scientific progress would slow down enormously. At WUWT the unqualified readers are supposed to find out themselves whether they are being conned or not. Even if they would do so: having every reader do a thorough review is wasteful; it is much more efficient to ask a few experts to first vet manuscripts.

Without peer review it would be harder for new people to get others to read their work, especially if they would make a spectacular claim and use unfamiliar methods. My colleagues will likely be happy to read my homogenization papers without peer review. Gavin Schmidt's colleagues will be happy to read his climate modelling papers and Michel Mann's colleagues his papers on climate reconstructions. But for new people it would be harder to be heard, for me it would be harder to be heard if I would publish something about another topic and for outsiders it would be harder to judge who is credible. The latter is increasingly important the more interdisciplinary sciences becomes.

Improving peer review

When I was dreaming of a future review system where scientific articles were all in one global database, I used to think of a system without journals or editors. The readers would simply judge the articles and comments, like on Ars Technica or Slashdot. The very active open science movement in Spain has implemented such a peer review system for institutional repositories, where the manuscripts and reviews are judged and reputation metrics are estimated. Let me try to explain why I changed my mind and how important editors and journals are for science.

One of my main worries for a flat database would be that there would be many manuscripts that never got any review. In the current system the editor makes sure that every reasonable manuscript gets a review. Without an editor explicitly asking a scientist to write a review, I would expect that many articles would never get a review. Personal relations are important.

Science is not a democracy, but a meritocracy. Just voting an article up or down does not do the job. It is important that this decision is made carefully. You could try to statistically determine which readers are good at predicting the quality of an article, where quality could be determined by later votes or citations. This would be difficult, however, because it is important that the assessment is made by people with the right expertise, often by people from multiple backgrounds; we have seen how much even something as basic as the scientific consensus on climate change depends on expertise. Try determining expertise algorithmically. The editor knows the reviewers.

While it is not a democracy, the scientific enterprise should naturally be open. Everyone is welcome to submit manuscripts. But editors and reviewers need to be trusted and level headed individuals.

More openness in publishing could in future come from everyone being able to start a "journal" by becoming editor (or better by organization a group of editors) and try to convince their colleagues that they do a good job. The fun thing about the annotation system is that you can demonstrate that you do a good job using existing articles and manuscripts.

This could provide real value for the reader. Not only would the reviews be visible, but it would also be possible to explain why an article was accepted, was it speculative, but really interesting if true (something for experts) or was it simply solid (something for outsiders). Which parts do the experts debate about. The debate would also continue after acceptance.

The code and the data of every "journal" should be open so that everyone can start a new "journal" with reviewed articles. So that when Heartland offers me a nice amount of dark money to start accepting WUWT-quality articles, a group of colleagues can start a new journal and fix my dark-money "mistakes", but otherwise have a complete portfolio from the beginning. If they would have to start from scratch that would be a large barrier to entry, which like the traditional system encourages sloppy work, corruption and power abuse.

Peer review is also not just for selecting articles, but also to help making them better. Theoretically the author can also ask colleagues to do so, but in practice reviewers are better in finding errors. Maybe because the colleagues who will put in most effort are your friends who have to same blind spots? These improvements of the manuscript would also be missing in a pure voting system of "finished" articles. Having a manuscript phase is helpful.

Finally, an editor makes anonymous reviews a lot less problematic because the editor could delete comment where the anonymity seduced people into inappropriate behavior. Anonymity could be abused to make false attacks with impunity. On the other hand anonymity can also provide protection in case of large power differences in case of real problems.

The advantage of internet publishing is that there is no need for an editor to reject technically correct manuscripts. If the contribution to science is small or if the result is very speculative and quite likely to be found to be wrong in future, the manuscript can still be accepted but simply be given a corresponding grade.

This also points to a main disadvantage of the current dead-tree-inspired system: you get either a yes or a no. There is a bit more information in the journal the author chooses, but that is about it. A digital system can communicate much more subtly with a prospective reader. A speculative article is interesting for experts, but may be best avoided by outsiders until the issues are better understood. Some articles mainly review the state-of-the-art, others provide original research. Some articles have a specific audience: for example the users of a specific dataset or model. Some articles are expected to be more important for scientific progress than others or discuss issues that are more urgent than others. And so on. This information can be communicated to the reader.

The nice thing about the open annotate system is that we can begin reviewing articles before authors start submitting their articles. We can simply review existing articles as well as manuscripts, such as the ones uploaded to ArXiv. The editors could reject articles that should not have been published in the traditional journals and accept manuscripts from archives. I would judge this assessment of a knowledgeable editor (team) more than the acceptance by a traditional journal.

In this way we can produce collections of existing articles. If the new system provides a better reviewing service to science, the authors at some moment can stop submitting their manuscripts to traditional journals and submit them directly to the editors of a collection. Then we have real grassroots scientific journals that serve science.

For colleagues in the communities it would be clear which of these collections have credibility. However, for outsiders we would also need some system that communicates this, which would traditionally be the role of publishing houses and the high barriers to entry. This could be assessed where collections have overlap. Preferably again by humans and not by algorithms. For some articles there may be legitimate reasons why there are differences (hard to assess, other topic of collection), for other articles an editor not having noticed problems may be a sign of bad editorship. This problem is likely not too hard, in a recent analysis of twitter discussions on climate change there was a very clear distinction between science and nonsense.

There is still a lot to do, but with the ease of modern publishing and the open annotate system a lot of software is already there. Larger improvements would be tools for editors to moderate review comments (or at least to collapse less valuable comments); Hypothes.is is working on it. A grassroots journal would need a grading system; standardized when possible. More practical tools would include some help in tracking the manuscripts under review and for sending reminders, and the editors of one collection should be able to communicate with each other. The grassroots journal should remain visible even if the editor team stops; that will need collaboration with libraries or science societies.

If we get this working

• we can say goodbye to frustrated reviewers (well mostly),
• goodbye to pay-walled journals in which publicly financed research is hidden for the public and many scientists alike and
• goodbye to wasting limited research money on monopolistic profits by publishing houses, while
• we can welcoming better review and selection and
• we are building a system that inherently allows for post-publication peer review.

What do you think?

Tweet

Related reading

There is now an "arXiv overlay journal", Discrete Analysis. Articles are published/hosted by ArXiv, otherwise traditional peer review. The announcement mentions three software initiative that make starting a digital journal easy: Scholastica, Episciences.org and Open Journal Systems.

Annotating the scholarly web

A coalition to Annotating All Knowledge A new open layer is being created over all knowledge

Brian A. Nosek and Yoav Bar-Anan describe a scientific utopia: Scientific Utopia: I. Opening scientific communication. I hope the ideas in the above post makes this transition possible.

Climate Feedback has started a crowed funding campaign to be able to review more media articles on climate science

Farewell peer reviewing

7 Crazy Realities of Scientific Publishing (The Director's Cut!)

Mapped: The climate change conversation on Twitter

I would trust most scientists to use annotation responsibly, but it can also be used to harass vulnerable voices on the web. Genius Web Annotator vs. One Young Woman With a Blog. Hypothesis is discussing how to handle such situations.

Nature Chemistry blog: Post-publication peer review is a reality, so what should the rules be?

Report from the Knowledge Exchange event: Pathways to open scholarship gives an overview of the different initiative to make science more open.

Magnificent BBC Reith lecture: A question of trust