German investigative reporter team uncovers large peer review scandal

The International Consortium of Investigative Journalists that also uncovered the Paradise Paper and Panama Papers investigated the world of predatory scientific journals and conferences. Most of the investigation was done by German journalists, where it has become a major news story that made the evening news.

The problem is much larger than I thought. In Germany it involves about five thousand scientists, about one percent sometimes use these predatory services. That is embarrassing and a waste of money.

While the investigators seem to understand scientific publishing well, it seems they do not understand science and the role of peer review in it. One piece of evidence is the preview picture of the documentary at the top of this article. It shows how the two reporters dressed up to present some nonsense at a fake conference. Presumably dressed up how they think scientists look like. Never seen such weird people at a real conference.

They may naturally claim they dressed like that to make it even more fake. But they also claim that no one in the audience noticed their presentation was fake, I guess they think so because they got a polite applause. That is not a good argument, everyone gets an applause, no matter how bad a presentation was.


A figure from the article that was published in the proceeding of the above fake conference with the title "Highly-Available, Collaborative, Trainable Communication - A Policy-Neutral approach". Clearly no one had a look at it before publishing. It got a "Best Presentation Award".

The stronger evidence that the reporters do not understand the way science works are the highly exaggerated conclusions they draw, which may lead to bad solutions. At the end of the above documentary (in German) the reporter asks: "Was wenn man keinen mehr glauben kann?", "What if you can no longer believe anyone?". Maybe the journalists forgot the ask the interviewed scientists to assess the bigger picture, which is what I will do in this post. There is no reason to doubt our scientific understand of the world because of this.

As an aside, the journalists of the International Consortium of Investigative Journalists are the good guys, but (Anglo-American) journalism that rejects objectivity is the bigger problem for the question what we can believe in than science.

Fortunately most of the reporting makes clear that the main driving force behind the problem is the publish-or-perish system that politicians and the scientific establishment have set up to micro-manage scientists. If you reward scientists for writing papers rather than doing science, they will write more papers, in the worst case in predatory journals.

Those in power will likely prefer to make the micromanagement more invasive and prescribe where scientists are allowed to publish. The near monopolistic legacy publishers, who are the only ones really benefiting from this dysfunctional system, will likely lobby in this direction.

Peer review

Many outside of science have unrealistic expectations of peer review and of peer reviewed studies. Peer review is just a filter that improves the average quality of the articles. Science does not deal in certainty (that is religion) and peer reviewed studies certainly do not offer certainty. The claims of single studies (and single scientists) may be better than a random blog post, but reliable scientific understanding should be based on all studies, preferably a few years old, and interpreted by many scientists.

This goes much against the mores of the news business, who focus on single studies that are just out, may add some personal interest by portraying single heroic scientists. The news likes spectacular studies that challenge our current understand and are thus the most likely studies to be wrong. If this mess the public sees were science we would not have much scientific progress.

As a consumer of science reporting I would much prefer to read overviews of how the understanding in a certain scientific community is and possibly how it has changed over the last years. It does not have to be recent to be new to me. There is so much I do not know.

---

the biggest threat to the proper public understanding of science is ... the lie we tell the public (and ourselves) that journal peer review works to separate valid and invalid science
Michael Eisen

---

Peer review is nothing more than that (typically) two independent scientists read the study, give their feedback on things that could be improved and advice the editor on whether the study is sufficiently interesting for the journal in question. Review is not there to detect fraud. Reviewers do not redo all experiments, do not repeat all calculations, do not know every related study that could shine a different light on the results. They just think that other scientists would be interested in reading the article. The main part of the checking, processing of the new information and weaving it into the scientific literature is performed after publication when scientists (try to) build on the work.

The documentary starts with someone with cancer who is conned into a treatment by a scrupulous producer pointing to their peer reviewed studies, partially published in predatory journals. They also criticize that articles from predatory journals were available in a database of a regulatory agency.

However, the treatment in question was not approved and the agency pointed out that they had not used these articles in their assessments. For these assessments scientists come together and discuss the entire literature and how convincing the evidence is in various aspects. These scientists know which journals are reliable, they read the studies and try to understand the situation. One of the interviewed scientists looked at one of the studies on this cancer treatment in a predatory journal and found several reasons why the journal should not have accepted it in the present form.

Also politicians would often like every scientific study to be so perfect that you do not need any expertise to interpret it and can directly use it for regulation. That is not how science works and also not what science was designed for. Science is not an enormous heap of solid facts. Science is a process where scientists gradually understand reality a little better.

Trying to get to the "ideal" of flawless and final studies would make do science much harder. Every scientists would have to be as smart and knowledgeable as the entire community working on something for years. Writing and reviewing scientific article would be so hard that scientific progress would come to a screeching halt. Especially new ideas would have no chance any more.

Conned scientists

Like most scientists I get multiple spam mails a day for fake scientific journals and conferences. Most of them have the quality of Nigerian Prince Spam. People say this kind of spam still exists because the spammers only want really stupid people to respond as they are the easiest to con.

Thus I had expected that people who take up such offers know what they are doing. Part of becoming a scientist is learning the publishing landscape of your field. But Open Access publishing reporter Richard Poynder mentioned several cases of scientists being honestly deceived, who tried to reverse their error when they noticed what happened.

The first researcher who contacted me realised something had gone wrong when the manuscript that he and his co-authors had submitted was returned to them with no peer review reports attached and no suggested changes. There was, however, a note to say that it had been accepted, and could they please pay the attached invoice. They later learned that the paper had already been published.

Quickly realising what had happened, and desperate to recover the situation, the authors agreed to pay the publisher the journal’s full [Author Processing Charges] (over $2,000) – not for publishing their paper, but for taking it down.

Apparently there are also predatory journals with names that are very similar to legitimate ones and a 1% error rate easily happens. I guess assessing the quality of journals can be harder in large fields and in case of interdisciplinary fields. If the first author selects a predatory journal, the co-authors may not have the overview of the journals in the other field to notice the problem.

We need to find a way help scientists who were honestly fooled and make it possible that the authors can retract their articles themselves. Otherwise they can be held hostage by the predatory publishers, which also funds the organised deception.

If the title of the real article is the same as the one of the predatory article it would be hard to put both on your CV or article list. Real publishers could be a bit more lenient there. A retraction notice of the predatory version in the acknowledgements of the real version should be "shameful" enough that people do not game the system, first publish predatory and then look for a real publisher.

Predators

If there is evidence that scientists purposefully publish in predatory journals or visit fake conferences that should naturally have consequences. That is wasting public money. One institute had 29 publications over a time span of ten years. There is likely a problem there.

It should also have consequences when scientists are in the editorial boards of such predatory journals. It may look nice on their CV to be editor, but editors should notice that they are not involved in the peer review or that it is done badly. It is hard to avoid the conclusion that they are aware that they are helping these shady companies. Sometimes these companies put scientists on their editorial boards without asking them. In that case you can expect a scientist to at least state on their homepage that they did not consent.

It is good to see that prosecutors are trying to take down some of these fake publishers. I wish them luck, although I expect this to be hard because it will be difficult to define how good peer review should work. Someone managed to get a paper published with the title "Get me off Your Fucking Mailing List". That would be a clear example of a fail and probably a case of one strike and you are out. At least scientifically, no idea about juridically. With more subtle cases you probably need to demonstrate that this happens more often. Climate "sceptics" occasionally manage to publish enormously bad articles in real scientific journals. That does not immediately make them predatory journals.

Changing publishing

In the past scientific articles were mostly published in paper journals to which academic libraries had subscriptions. This made it hard for the public and many scientists to read scientific articles, especially for scientists from the global South, but I also cannot read articles in one of the journals I publish in regularly myself.

Nowadays this system is no longer necessary as journals can be published online. Furthermore, the legacy system is made for monopolies: a reader needs a specific article and an author needs a journal that most scientists subscribe to. As society replaces morality with money and as the publishing industry is concentrating and clearly prioritizes profits over being a good member of the scientific community subscription prices have gone up and service has gone down. As an example of the former, Elsevier has a profit margin of 30 to 50 percent. As an example of the latter, in one journal I unfortunately publish in the manuscript submission system is so complicated that you have to reserve almost a full working day to submit a manuscript.

The hope of the last decade was that a new publishing model would break open the monopoly: open access publishing. In this model articles are free to read and in most cases the authors fund the journals. This reduces the monopoly power of the journals. Readers can read the articles they need and authors can be sure their colleagues can read the article. However, scientists want to publish in journals with a good reputation, which takes years if not decades to build up and still produces a quite strong monopoly situation.

This has resulted in publishing fees of several thousands of Euro for the most prestigious open access journals. In this way these journals are open to read, but no longer open to publish for many researchers. These journals drain a lot of resources that could have been used for research; likely more than the predatory publishers ever will. My guess would be that the current publishing system is 50 to 90 percent too expensive; the predatory journals have less than 1 percent of the market.

The legacy publishers defend their profits and bad service with horror stories about predatory open access journals. They prefer to ignore all the high quality open access journals. This investigative story unfortunately feeds this narrative.

Bad solutions

The Austrian national science foundation (FWF) has found a way to make the situation worse. They want make sure that the scientists they fund will only publish in a list of known good quality open access journals, for example in the Directory of Open Access Journals. That sounds good, but if all science foundations would adopt this policy it would become nearly impossible to start new scientific journals and the monopolies would get stronger again.

[UPDATE The German Alliance of Scientific Organizations fortunately states that journal selection is part of the freedom of science. They furthermore state that the quality of a study does not depend on where it is published and want to help scientists with training and information persons. They see a key role for the Directory of Open Access Journals (DOAJ).]

I just send a nice manuscript to a new journal, which has no real reputation yet. Its topics fits very well to my work, so I am happy my colleagues started this journal. I did my due diligence, know several people on the editorial board as excellent researchers and even looked through a few published articles. The same publisher has many good journals and journal is by now also listed in the Directory of Open Access Journals (DOAJ). The DOAJ was actually very quick and already listed this journal after only publishing 11 articles. But getting those first 11 would be hard if the FWF policy wins out.

The opposite model is to create a black list. This has less problems, but it is quite hard to determine which journals are predatory. There used to be a list of predatory journals by Jeffrey Beall, but he had to stop because of legal threads to his university by the predatory publishers. There were complaints that this list discriminated against journals from developing countries. True or not, this illustrates how hard it is to maintain such a list. There is now, oh irony, a pay-walled version of such a list with predatory journals. The subscriptions should probably pay for the legal risks.

Changing publishing

A good solution would be to review the articles after publication. This would allow researchers to update their assessments when evidence from newer studies come in and we understand the older studies better. PubPeer is a system to do this post-publication peer review, but it mostly has reviews for flawed papers and thus does not give a good overview over the scientific literature.

F1000 Prime is an open access journal with post publication review. I know of two more complete post-publication review systems: The Self-Journals of Science and recently Peeriodicals. Here every scientist can start a journal, collect the articles that are worthwhile and write something about them. The more scientists endorse an article, the more influential it is. In these systems I miss reviews of article are not that important, but are valid and they may still be informative for some. Furthermore, I would expect that the review would need to be organized more formally to be seen as worthy successors of the current quality control system.

That is what I am trying to build up at the moment and I have started a first such "grassroots journal" for my own field to show how the system would work. I expect that the system will be superior because these "grassroots journals" do not publish the articles themselves, only review them, and thus can assess all articles in one field at one place, while traditionally articles are spread over many journals. The quality of the reviews will be better because it uses a post-publication review model. The reviews are more helpful to the readers because they are published themselves and quantify in more detail what is good about an article. As such it performs the role of a supervisor in finding one's way in the scientific literature.

You get a similar effect from the always up-to-date review paper on sea surface temperature proposed and executed by my colleagues John Kennedy. This makes it easy for others to contribute, while having versioning and attribution. There is naturally less detail per article that is reviewed.

Changing the system

But also a better reviewing system cannot undo the damage of the fake competitive system currently used to fund scientific research.

Volker Epping, president of the University of Hannover, stated: "The pressure to publish is enormous. Problems are inherent to the system." I would even argue: Given the way the system is designed, it is a testament of the dedication of the scientists that it still works so well.

It is called "competitive", but researchers are competing to get their colleagues to approve the funding their research. There is no real competition because there is no real market. If you did a good job, there are no customers that reward you for this. In the best case the rewards come as new funding decided by people who have no skin in the game. People who have no incentive to make good funding decisions. Given that situation, it is amazing that scientists still spend time in making good peer reviews of research proposals and show dedication comparing them with each other to decide what to fund.

My proposal would be to return to the good old days. Give the funding to the universities, which give it to the professors, which allocate it to what they, as the most informed experts, think is interesting research, which furthers their reputation. Professors have skin in the game, their reputation is on the line, and they will invest the limited funds where they expect to get most benefits. In the current system there is no incentive to set priorities, submitting more research proposals has no downsides for them beyond the time it takes to write them. One of the downsides of this model for science is that the best researchers are not doing research, but are writing research proposals.

A compromise could be to limit the number of projects a science foundation funds per laboratory. The Swiss Science Foundation uses this model.

The old and hopefully future system also allows for awarding permanent positions to good researchers. Now most researchers are on short-term contracts because the project funding does not provide stable funding. With these better labour conditions one could attract much better researchers for the same salary.

Because project science requires so many peer reviews (of research proposals and of a bloated number of articles) a lot of time is wasted. (This waste is again much bigger than that of the predatory publishers.) This invites the reviewers to use short-cuts and not assess how good a scientist is and instead assess how many articles they write and how high the prestige is of the journals the articles appear in (bibliometric measures). Officially this system is illegal in Germany, the ethics rules of the German Science Foundation forbid judging researchers and small groups on their bibliometric measures, but it still happens.

My expectation is that without the publish-or-perish system scientific progress would go much faster and we certainly would not have the German public being shocked to learn about predatory publishers.

I hope the affair will inspire journalists to inform the public better on how science works and what peer review is and is not.

Related reading

Investigation of predatory publishing

English summary by the International Consortium of Investigative Journalists (ICIJ): New international investigation tackles ‘fake science’ and its poisonous effects.

A critical comment in English on the affair: Beyond #FakeScience: How to Overcome Shallow Certainty in Scholarly Communication. With many (mostly German) links to news sources.

The newspaper Indian Express (the Indian partner of the ICIJ): Inside India’s fake research paper shops: pay, publish, profit. Despite UGC blacklist, hundreds of ‘predatory journals’ thrive, cast shadow on quality of faculty and research nationwide.

Comment on the investigation: Predatory Open Access Journals: Is Open Peer Review Any Help? I think it would help, but there are also commercial firms where you can buy peer reviews (next to copy editing, statistical analysis and writing complete articles and theses).

The Bern university library feels criteria for black lists are not transparent and making them puts much power into the hands of a commercial company: On the topic of #predatoryjournals - are there black lists and how reliable are they? In German: Zum Thema #predatoryjournals – gibt es schwarze Listen und wie verlässlich sind sie?

Investigation of scientists of the City University of New York publishing in predatory journals.

Overview of the investigation by the ICIJ in German: Overview of the investigative project in German.

Q&A of ICIJ in German: #FakeScience - Fragen und Antworten.

Why so many researchers use dubious ways to publish (in German). Warum so viele Forscher auf unseriösem Weg publizieren. Volker Epping, president of the University of Hannover: "The pressure to publish is enormous. Problems are inherent to the system." I would even argue: Given the way the system is designed, it is a testament of the dedication of the scientists that it still works so well.

Video of a comment on the affair by Svea-Eckert in German. Like other jouralists, I think she is wrong about the implications.

Peer review

The first grassroots scientific journal, which I hope will inspire the post-publication review system of the future.

An always up-to-date review paper by John Kennedy, Elizabeth Kent on GitHub: A review of uncertainty in in situ measurements and data sets of sea-surface temperature. You can use bug reports and pull requests to add to the text.

Separation of feedback, publishing and assessment of scientific studies.

Separation of review powers into feedback and importance assessment could radically improve peer review. Grassroots scientific journals.

Publish or perish is illegal in Germany, for good reason. German science has another tradition, trusting scientists more and focusing on quality. This is expressed in the safeguards for good scientific practice of the German Science Foundation (DFG). It explicitly forbids the use of quantitative assessments of articles.

The value of peer review for science and the press. Is it okay to seek publicity for a work that is not peer reviewed? Should a journalist only write about peer-reviewed studies? Is peer review gate keeping? Is peer review necessary?

Peer review helps fringe ideas gain credibility.

Three cheers for gatekeeping.

Did Isaac Newton Need Peer Review? Scholarly Journals Swear By This Practice of Expert Evaluation. But It’s a New Phenomenon That Isn’t the Only Way To Establish the Facts.

Think. Check. Submit. How to recognise predatory publishers before you submit your work.

The fight for the future of science in Berlin

A group of scientists, scholars, data scientists, publishers and librarians gathered in Berlin to talk about the future of research communication. With the scientific literature being so central to science, one could also say the conference was about the future of science.

This future will be more open, transparent, findable, accessible, interoperable and reusable.


The open world of research from Mark Hooper on Vimeo.

Open and transparent sounds nice and most seem to assume that more is better. But it can also be oppressive, help the powerful with the resources to mine the information efficiently.

This is best known when it comes to government surveillance, which can be dangerous; states are powerful and responsible for the biggest atrocities in history. The right to vote in secret, to privacy, to organize and protections against unreasonable searches are fundamental protections against power abuse.

Powerful lobbies and political activists abuse transparency laws to harass inconvenient science.

ResearchGate, Google Scholar profiles and your ORCID ID page contribute to squeezing scientists like lemons by prominently displaying the number of publications and citations. This continual pressure can lead to burnout, less creativity and less risk taking. It encourages scientists to pick low hanging fruits rather than do those studies they think would bring science forward the most. Next to this bad influence on publications many other activities, which are just as important for science, suffer from this pressure. Many good-willing people were trying to solve this by also quantifying these activities. But in doing so add more lemon presses.


That technology brings more surveillance and detrimental micro-management is not unique to science. The destruction of autonomy is a social trend, that, for example, also affects truckers.

Science is a creative profession (even if many scientists do not seem to realise this). You have good ideas when you relax under the shower, in bed with fever or on a hike. The modern publish or perish system is detrimental to cognitive work. Work that requires cognitive skills is performed worse if you pressure people, it needs autonomy, mastery and purpose.

Scientists work on the edge of what is known and invariably make mistakes often. If you are not making mistakes you are not pushing your limits. This needs some privacy because unfortunately making mistakes is not socially acceptable for adults.

Chinese calligraphy with water on a stone floor. More ephemeral communication can lead to more openness, improve the exchange of views and produce more quality feedback.

Also later on in the process the ephemeral nature of a scientific talk requires deep concentration from the listener and is a loss for people not present, but it is also a feature early in a study. Without the freedom to make mistakes there will be less exiting research and slower progress. Scientists are also humans and once an idea is fixed on "paper" it becomes harder to change, while the flexibility to update your ideas to the evidence is important and likely needed in early stages.

These technologies also have real benefits, for example, make it easier to find related articles by the same author. A unique researcher identifier like ORCID especially helps when someone changes their name or in countries like China where one billion people seem to share about 1000 unique names. But there is no need for ResearchGate to put the number of publications and citations in huge numbers on the main profile page. (The prominent number of followers on Twitter profile pages also makes it less sympathetic in my view and needlessly promotes competition and inequality. Twitter is not my work, artificial competition is even more out of place.)

Open Review is a great option if you are confident about your work, but fear that reviewers will be biased, but sometimes it is hard to judge how good your work is and nice to have someone discretely point to problems with your manuscript. Especially in interdisciplinary work it is easy to miss something a peer review would notice, while your network may not include someone from another discipline you can ask to read the manuscript.

Once an article, code or dataset is published, it is free game. That is the point where I support Open Science. For example, publishing Open Access is better than pay-walled. If there is a reasonable chance of re-use publishing data and code helps science progress and should be rewarded.

Still I would not make a fetish out of it; I made the data available for my article on benchmarking homogenisation algorithms. This is an ISI highly-cited article, but I only know of one person having used the data. For less important papers publishing data can quickly be additional work without any benefits. I prefer nudging people towards Open Science over making it obligatory.

The main benefactor of publishing data and code is your future self, no one is more likely to continue your work. This should be an important incentive. Another incentive are Open Science "badges": icons presented next to the article title indicating whether the study was preregistered and provides open data and open materials (code). The introductions of these badges in the journal "Psychological Science" increased the percentage of articles with available data quickly to almost 40%.

The conference was organised by FORCE11, a community interested in future research communication and e-scholarship. There are already a lot of tools for the open, findable and well-connected world of the future, but their adoption could go faster. So the theme of this year's conference was "changing the culture".

Open Access

Christopher Jackson; on the right. (I hope I am allowed to repeat his joke.)

A main address was by Christopher Jackson. He has published over 150 scientific articles, but only became aware of how weird the scientific publishing system is when he joined ResearchGate, a social network for scientists, and was not allowed to put many of his articles on it because the publishers have the copy rights and do not allow this.

The frequent requests for copies of his articles on Research Gate also created an awareness how many scientists have trouble accessing the scientific literature due to pay-walls.

Another key note speaker, Diego Gómez, was threatened with up to eight years in jail for making scientific articles accessible. His university, Universidad del Quindio in Costa Rica, spends more on licenses for scientific journals ($375,000) than on producing scientific knowledge themselves ($253,000).



The lack of access to the scientific literature makes research in poorer countries a lot harder, but even I am regularly not able to download important articles and have to ask the authors for a copy or ask our library to order a photocopy elsewhere, although the University of Bonn is not a particularly poor university.

Also non-scientists may benefit from being able to read scientific articles, although when it is important I would prefer to consul an expert over mistakenly thinking I got the gist of an article in another field. Sometimes a copy of the original manuscript is found on one of the authors homepages or a repository. Google (Scholar) and the really handy browser add-on unpaywall can help find those using the Open Access DOI database.

Also sharing passwords and Sci-Hub are solutions, but illegal. The real solutions to making research more accessible are Open Access publishing and repositories for manuscripts. By now about half of the recently published articles are Open Access and in this pace all articles would be Open Access by 2040. Interestingly the largest fraction of the publicly available articles does not have an Open Access license, also called bronze Open Access. This means that the download possibility could also be revoked again.

The US National Institutes of Health and the European Union mandate that its supported research will be published Open Access.

A problem with Open Access journals can be that some are only interested in the publication fees and do not care about the quality. These predatory journals are bad for the reputation of real Open Access journals, especially in the eyes of the public.

I have a hard time believing that the authors do not know that these journals are predatory. Next to the sting operations to reveal that certain journals will publish anything, it would be nice to also have sting predatory journals that openly email the authors that they will accept any trash and see if that scares away the authors.

Jeffrey Beall used to keep a list of predatory journals, but had to stop after legal pressure from these frauds. The publishing firm Cabell now launched their own proprietary (pay-walled) blacklist, which already has 6000 journals and is growing fast.

Preprint repositories

Before a manuscript is submitted to a journal, the authors naturally still have the copy rights. They can thus upload the manuscript to a database, so-called preprint or institutional repositories. Unfortunately some publishers say this constitutes publishing a manuscript and they refuse to publish it because it is no longer new. However, most publishers accept the publications of the manuscript as it was before submission. A smaller part is also okay with the final version being published on the author's homepages or repositories.

Where a good option for an Open Access journal exists we should really try to use it. Where it is allowed, we should upload our manuscripts to repositories.

Good news for the readers of this blog is that a repository for the Earth Sciences was opened last week: EarthArXiv. This fall, the AGU will also demonstrate its preprint repository at the AGU Fall meeting. For details see my previous post.  EarthArXiv already has 15 climate related preprints.

This November also a new OSF ArXiv has started: MarXiv, not for Marxists, but for the marine-conservation and marine-climate sciences.

When we combine the repositories with peer review organised by the scientific community itself, we will no longer need pay-walling scientific publishers. This can be done in a much more informative way than currently where the reader only knows that the paper was apparently good enough for the journal, but not why it is a good article, not how it fits in the (later published) literature. With Grassroots scientific publishing we can do a much better job.

One way the reviews at a Grassroots journal can be better is by openly assessing the quality of the work. Now all we know is that the study was sufficiently interesting for some journal at that time for whatever reason. What I did not realise before Berlin is that this wastes a lot of time reviewing. Traditional journals waste resources on manuscripts, which are valid, but are rejected because they are seen as not important enough for the journal. For example, Frontiers reviews 2.4 million manuscripts and has to bounce about 1 million valid papers.

On average scientists pay $5,000 per published article. This while scientists do most of the work for free (writing, reviewing, editing) and while the actual costs are a few $100. The money we save can be used for research. In the light of these numbers it is actually amazing that Elsevier only makes a profit of 35 to 50%. I guess their CEO's salary eats into the profits.

Preprints would also have the advantage of making studies available faster. Open Access makes text and data mining easier, which helps in finding all articles on molecule M or receptor R. First publishers are using Text mining and artificial intelligence to suggest suitable peer reviewers to their editors. (I would prefer editors who know their field.) It would also help in detecting plagiarism and even statistical errors.

(Before our machine overlords find out, let me admit that I did not always write the model description of the weather prediction model I used from scratch.)

Impact factors

Another issue Christopher Jackson highlighted is the madness of the Journal Impact Factors (JIF or IF). They measure how often an average article in a journal is cited in the first two or five years after publication. They are quite useful for librarians to get an overview over which journals to subscribe to. The problem begins when this impact factor is used to determine the quality of a journal or the articles in it.

How common this is, is actually something I do not know. For my own field I would think I have a reasonable feeling about the quality of the journals, which is independent of the impact factor. More focussed journals tend to have smaller impact factors, but that does not signal that they are less good. Boundary Layer Meteorology is certainly not worse than the Journal of Geophysical Research. The former has in Impact Factor of 2.573, the latter of 3.454. If you made a boundary layer study it would be madness to then publish it in a more general geophysical journal where the chance is smaller that relevant colleagues will read it. Climate journals will have higher impact factors than meteorological journals because meteorologists mainly cite each other, while many sciences build on climatology. When the German meteorological journal MetZet was still a pay-wall journal it had a low impact factor because not many outside of Germany had a subscription, but the quality of the peer review and the articles was excellent.

I would hope that reviewers making funding and hiring decisions know the journals in their field and take these kind of effects into account and read the articles itself. The [[San Francisco Declaration on Research Assessment]] (DORA) rejects the use of the impact factor. In Germany it is officially forbidden to judge individual scientists and small groups based on bibliographic measures such as the number of articles times the impact factor of the journals. Although I am not sure if everybody knows this. Imperial College recently adopted similar rules:

“the College should be leading by example by signalling that it assesses research on the basis of inherent quality rather than by where it is published”
“eliminate undue reliance on the use of journal-based metrics, such as JIFs, in funding, appointment, and promotion considerations”

The relationship between the number of citations an article can expect and the impact factor is weak because there is enormous spread. Jackson showed this figure.



This could well be a feature and not a bug. We would like to measure quality, not estimate the (future) number of citations of an article. For my own articles, I do not see much correlation between my subjective quality assessment and the number of citations. Which journal you can get into may well be a better quality measure than individual citations. (The best assessment is reading articles.)

The biggest problem is when the journals, often commercial entities, start optimising for the number of citations rather than quality. There are many ways to get more citations, a higher impact factor, than making the best possible quality control. An article that reviews the state of the scientific field typically get a lot of citations, especially if writing by the main people in the field. Nearly every article will mention it in the introduction. Review papers are useful, but we do not need a new one every year. Articles with many authors typically get more citations. Articles on topics many scientists work on will get more citations. For Science and Nature it is important to get coverage in the main stream press, which is also read by scientists and leads to more citations.

Reading articles is naturally work. I would suggest to reduce the number of reviews.

Attribution, credit

Traditionally one gets credit for scientific work by being author of a scientific paper. However, with increased collaboration and interdisciplinary work author lists have become longer and longer. Also the publish or perish system likely contributed: outsourcing part of the work is often more efficient than doing it yourself, while the person doing a small part of the analysis is happy to have another paper on their publish or perish list.

What is missing from such a system is getting credit for a multitude of other import tasks. How does one value non-traditional output items supplied by researchers: code, software, data, design, standards, models, MOOC lectures, newspaper articles, blog posts, community engaged research and citizen science? Someone even mentioned musicals.

A related question is who should be credited: technicians, proposal writers, data providers? As far as I know it would be illegal to put people in such roles in author list, but they do work that is important, needs to be done and thus needs to be credited somehow. A work-around is to invite them to help in editing the manuscript, but it would be good to have systems where various roles are credited. Designing such a system is hard.

One is temped to make such a credit system very precise, but ambiguity also has its advantages to deal with the messiness of reality. I once started a study with one colleague. Most of this study did not work out and the final article was only about a part. A second colleague helped with that part. For the total work the first colleague had done more work, for the part that was published the second one. Both justifiably found that they should be second author. Do you get credit for the work or for the article?

Later the colleague who had become third author of this paper wrote another study where I helped. It was clear that I should have been the second author, but in retaliation he made me the third author. The second author wrote several emails that this was insane, not knowing what was going on, but to no avail. A too precise credit system would leave no room for such retaliation tactics to clear the air for future collaborations.

In one session various systems of credit "badges" were shown and tried out. What seemed to work best was a short description of the work done by every author, similar to a detailed credit role at the end of a movie.

This year a colleague wrote on a blog that he did not agree with a sentence of an article he was author of. I did not know that was possible; in my view authors are responsible for the entire article. Maybe we should also split up the authors list in authors who guarantee with their name and reputation for the quality of the full article and honorary authors who only contributed a small part. This colleague could then be a honorary author.

LindedIn endorsements were criticised because they are not transparent and they make it harder to change your focus because the old endorsements and contacts stick.

Pre-registration

Some fields of study have trouble replicating published results. These are mostly empirical fields where single studies — to a large part — stand on their own and are not woven together by a net of theories.

One of the problems is that only interesting findings are published and if no effect is found the study is aborted. In a field with strong theoretical expectations also finding no effect when one is expected is interesting, but if no one expected a relationship between A and B, finding no relationship between A and B is not interesting.

This becomes a problem when there is no relationship between A and B, but multiple experiments/trails are made and some will find a fluke relationship by chance. If only those get published that gives a wrong impression. This problem can be tackled by registering trails before they are made, which is becoming more common in medicine.

A related problem is p-hacking and hypothesis generation after results are known (HARKing). A relationship which is statistically significant if only one outlier were not there, makes it tempting to find a reason why the outlier is a measurement error and should be removed.

Similarly the data can be analyses in many different ways to study the same question, one of which may be statistically significant by chance. This is also called "researcher degrees of freedom" or "the garden of forking paths". The Center for Open Science has made a tool where you can pre-register your analysis before the data is gathered/analysed to reduce the freedom to falsely obtain significant results this way.



A beautiful example of the different answers one can get analysing the same data for the same question. If found this graph via a FiveThirtyEight article, which is also otherwise highly recommended: "Science Isn’t Broken. It’s just a hell of a lot harder than we give it credit for."

These kind of problems may be less severe in natural sciences, but avoiding them can still make the science more solid. Before Berlin I was hesitant about pre-registering the analysis because in my work every analysis is different, which makes is harder to know in detail in advance how the analysis should go; there are also valid outlier that need to be removed, selecting the best study region needs a look at the data, etc.

However, what I did not realise, although quite trivial, is that you can do the pre-registered analysis, but also additional analysis and simply mark them as such. So if you can do a better analysis after looking at the data, you can still do so. One of the problems of pre-registering is that quite often people did not do the analysis in the same way and that reviewers mostly do not check this.

In the homogenisation benchmarking study of the ISTI we will describe the assessment measures in advance. This is mostly because the benchmarking participants have a right to know how their homogenisation algorithms will be judged, but it can also be seen as pre-registration of the analysis.

To stimulate the adoption of pre-registration, the Center for Open Science has designed Open Science badges, which can be displayed with the articles meeting the criteria. The pre-registration has to be done at an external site where the text cannot be changed afterwards. The pre-registration can be kept undisclosed for up to two years. To get things started they even award 1000 prices of $1000 for pre-registered studies.

The next step would be journals that review "registered reports", which are peer reviewed before the results are in. This should stimulate the publication of negative (no effect found) results. (There is still a final review when the results are in.)

Quick hits

Those were the main things I learned, now some quick hits.

With the [[annotation system]] you can add comments to all web pages and PDF files. People may know annotation from Hypothes.is, which is used by ClimateFeedback to add comments to press articles on climate change. A similar initiative is PaperHive. PaperHive sells its system as collaborative reading and showed an example of students jointly reading a paper for class, annotating difficult terms/passages. It additionally provides channels for private collaboration, literature management and search. It has also already been used for the peer review (proof reading) of academic books. They now both have groups/channels to allow groups to make or read annotations, as well as private annotations, which can be used for your own paper archive. Web annotations aimed at the humanities are made by Pund.it.

Since February this year, web annotation is a World Wide Web (W3C) standard. This will hopefully mean that web browsers will start adding annotation in their default configuration and it will be possible to comment every homepage. This will likely lead to public annotation streams going down to the level of YouTube comments. Also for the public channel some moderation will be needed, for example to combat doxing. PaperHive is a German organisation and thus removes hate speech.

Peer Community In (PCI) a system to collaboratively peer review manuscripts that can later be send to an official journal.

The project OpenUp studied a large number of Open Peer Review systems and their pros and cons.

Do It Yourself Science. Not sure it is science, but great when people are having fun with science. When the quality level is right, you could say it is citizen science led by the citizens themselves. (What happened to the gentlemen scientists?)

Philica: Instant academic publishing with transparent peer-review.

"Standards are like toothbrushes; everyone likes the idea of them, but everyone wants to use their own" #FORCE2017 @Metadata2020

— Juan Pablo Alperin (@juancommander) 26 October 2017

Unlocking references from the literature: The Initiative for Open Citations. See also their conference abstract.

I never realised there was an organisation behind the Digital Object Identifiers for scientific articles: CrossRef. It is a collaboration of about eight thousand scientific publishers. For other digital sources there are other organisation, while the main system is run by the international DOI Foundation. The DOIs for data are handled, amongst others, by DataCite. CrossRef is working on a system where you can also see the webpages that are citing scientific articles, what they call "event data". For example, this blog has cited 142 articles with a DOI. CrossRef will also take web annotations into account.

Climate science was well represented at this conference. There were posters on open data for the Southern Ocean and on the data citation of the CMIP6 climate model ensemble. Shelley Stall of AGU talked about making FAIR and Open data the default for Earth and space science. (Et moi.)



In the Life Sciences they are trying to establish "micro publications", the publication of a small result or dataset, several of which can then later be combined with a narrative into a full article.

A new Open Science Journal: Research Ideas and Outcomes (RIO), which publishes all outputs along the research cycle, from research ideas, proposals, to data, software and articles. They are interested in all areas of science, technology, humanities and the social sciences.

Collaborative writing tools are coming of age, for example, Overleaf for people using LaTex. Google Docs and Microsoft Word Live also do the trick.

Ironically Elsevier was one of the sponsors. Their brochure suggests they are ones of the nice guys serving humanity with cutting edge technology.

The Web of Knowledge/Science (a more selective version of Google Scholar) moved from Thomson Reuters to Clarivate Analytics, together with the Journal Citation Reports that computes the Journal Impact Factors.

Publons has set up a system where researchers can get public credit for their (anonymous) peer reviews. It is hoped that this stimulates scientists to do more reviews.

As part of Wikimedia, best known for Wikipedia, people are building up a multilingual database with facts: wikidata. Like in Wikipedia volunteers build up the database and sources need to be cited to make sure the facts are right. People are still working on software to make contributing easier for people who are not data scientists and do not dream of the semantic web every night.

Final thoughts

For a conference about science, there was relatively little science. One could have made a randomized controlled trial to see the influence of publishing your manuscript on a preprint server. Instead the estimated larger number of citations for articles also submitted to ArXiv (18%) was based on observational data and the difference could be that scientists put more work in spreading their best articles.

The data manager at CERN argued that close collaboration with the scientists can help in designing interfaces that promote the use of Open Science tools. Sometimes small changes produce large increases in adoption of tools. More research into the needs of scientists could also help in creating the tools in a way that they are useful.

Related reading, resources

The easiest access to the talks of the FORCE2017 conference is via the "collaborative note taking" Google Doc

Videos of last year's FORCE conference

Peer review

The Times Literary Supplement: Peer review: The end of an error? by ArXiving mathematician Timothy Gowers

Peer review at the crossroads: overview over the various open review options, advantages and acceptance

Jon Tennant and many colleagues: A multi-disciplinary perspective on emergent and future innovations in peer review

My new idea: Grassroots scientific publishing

Pre-prints

The Earth sciences no longer need the publishers for publishing
 
ArXivist. A Machine Learning app that suggest the most relevant new ArXiv manuscripts in a daily email

The Stars Are Aligning for Preprints. 2017 may be considered the ‘year of the preprint

Open Science

The State of OA: A large-scale analysis of the prevalence and impact of Open Access articles

Open Science MOOC (under development) already has an extensive resources page

Metadata2020: Help us improve the quality of metadata for research. They are interested in metadata important for discoverability and reuse of data

‘Kudos’ promises to help scientists promote their papers to new audiences. For example with plain-language summaries and tools measure which dissemination actions were effective

John P. A. Ioannidis and colleagues: Bibliometrics: Is your most cited work your best? Survey finds that highly cited authors feel their best work is among their most cited articles. It is the same for me, still looking at all articles it is not a strong correlation

Lorraine Hwang and colleagues in Earth and Space Science: Software and the scientist: Coding and citation practices in geodynamics, 2017

Neuroskeptic: Is Reproducibility Really Central to Science?

The Trump administration proposes a new scientific method just for climate studies

What could possibly go wrong?

[[Scott Pruitt]] is the former Oklahoma Attorney General who copied and pasted letters for pro-pollution lobbyists onto his letter head. Much of his previous work was devoted to suing the EPA. Now he works for the big money donors as head of the EPA.  This Scott Pruitt is allegedly working on formulating a new scientific method to be used for studying climate change alone. E&E News just reported that this special scientific method will use "red team, blue team" exercises to conduct an "at-length evaluation of U.S. climate science."

Let's ignore that it makes no sense to speak of US climate science when it comes to the results. Climate science is the same in every country. There tends to be only one reality.

Previously [[Rick Perry]], head of the Department of Energy (DOE) who campaigned on closing the DOE before he knew what it does, had joined the group calling to replace the scientific method with a Red Team Blue Team exercise.



A Red Team is supposed to challenge the claims of the Blue Team. It is an idea from hierarchical organisations, like the military and multinationals, where challenging the orthodoxy is normally not appreciated and thus needs to be specially encouraged when management welcomes it.

Poking holes is our daily bread

It could naturally be that the climate "sceptics" do not know that challenging other studies is build in into everything scientists do; they do not give the impression to know science that well. In their Think Tanks and multinational corporations they are probably happy to bend the truth to get ahead. They may think that that is how science works and they may not able to accept that a typical scientist is intrinsically motivated to figure out how reality works.

People driven by greed are unable to comprehend any other motivation. Similarly, habitual liars assume everyone else lies just like they do.

— Joe Daniel (@Bikes4bacon) 1 July 2017

At every step of a study a scientist is aware that at the end it has to be written up very clearly to be criticised by peer reviewers before publication and by any expert in the field after publication. That people will build on the study and in doing so may find flaws. Scientific claims should be falsifiable, one should be able to show them wrong. The main benefit of this is that it forces scientists to very clearly describe the work and make it vulnerable to attack.

The first time new results are presented is normally in a working group seminar where the members of the Red Team are sitting around the table, ask specific questions during the talk and criticise the main ideas after the talk. These are scientists working with similar methods, but also ones who work on very different problems. All and especially the group leaders have an interest in defending the reputation of the group and making sure no nonsense spoils it.

The results are normally also presented at workshops, conferences and invited talks at other groups. At workshops leading experts will be there working on similar problems, but with a range of different methods and backgrounds. At conferences and invited talks there are in addition also many scientists from adjacent fields in the audience or scientists working with similar methods on other problems. A senior scientist will get blunt questions after the talk if anything is wrong with it. Younger scientists will get nicer questions in public and the blunt ones in private.

An important Red Team consists of your co-authors. Modern science is mostly done in teams. That is more efficient, reduces the chances of rookie errors and very easy due to the internet. The co-authors guarantee with their reputation for the quality of the study, especially for the part where they have expertise.

None of these steps are perfect and journalists should get away from their single-study fetish. But together these steps ensure that the quality of the scientific literature as a whole is high.

(It is actually good that none of these steps are perfect. Science works on the boundary of what is known, scientists that do not make errors are not pushing themselves enough. If peer review would only pass perfect articles that would be highly inefficient and not much would be published, it normally takes several people and studies until something is understood. It is helpful that the scientific literature is high quality, it does not need to be perfect.)

Andrew Revkin should know not to judge the quality of science by single papers or single scientists, that peer review does not need to be perfect and did not exist for most of the scientific era. But being a false balance kind of guy he regrettably uses "Peer review is often not as adversarial as intended" as argument to see merit in a Red Team exercise. While simultaneously acknowledging that "All signs point to political theater"

Red Team science

An optimistic person may think that the Red Team proposal of the Trump administration will follow the scientific method. We already had the BEST project of the conservative physics professor Richard Muller. BEST was a team of outside people have a look at the warming over land estimated from weather station observations. This project was funded in part by the Charles G. Koch Foundation. the Heartland Institute, hard core deniers funded by Koch Brother organisations.

The BEST project found that the previous scientific assessments of the warming were right.



The BEST project is also a reason not to be too optimistic about Pruitt's proposal. Before BEST published their results mitigation sceptics were very enthusiastic about their work and one of their main bloggers, Anthony Watts, claimed that their methods were so good and he would accept the outcome no matter the result. That changed when the result was in.

Judith Curry was part of BEST, but left before she would have had to connect her name with the results. Joseph Majkut of Niskanen Center, who wrote an optimistic Red Team article, claims there were people who changed their minds due to BEST, but did not give any examples yet.

It also looks as if BEST was punished for the result that was inconvenient for the funders. The funders are apparently no longer interest in studying the quality of climate observations. Berkeley Earth now mainly works on air pollution. While BEST did not even look at the largest part of the Earth yet: the oceans. The nice thing of being funded by national science foundations is that they care about the quality of the work, but not the outcomes.

If coal or oil corporations thought there was a minute possibility that climate science was wrong, they would fund their own research. Feel free to call that Red Team research. That they invest in PR instead shows how confident they are that the science is right. Initially Exxon did fund research, when it became clear climate change was a serious risk they switched to PR.

Joseph Majkut thinks that a well-executed Red Team exercise could convince people. In the light of the BEST project, the corporate funding priorities and the behaviour of mitigation sceptics in the climate "debate", I am sceptical. People who did not arrive at their position because of science will not change their position because of science.

Washington Republicans will change their mind when the bribes, aka campaign contributions, of the renewable energy sector are larger than those of the fossil fuel sector. Or when the influence of money is smaller than that of the people, like in the good old days.

Science lives on clarity

As a scientist, I would suggest just wait and see at this time. Let the Trump administration make a clear plan for this new scientific method. I am curious.

Let them tell us how they will select the members of the Red Team. Given that scientists are always critiquing each others work, I am curious how they plan to keep serious scientists out of their Red Team. I would be happy to join, there is still a lot of work to do on the quality of station data. Scientific articles typically end with suggestions for future research. That is the part I like writing the most.

Because the Trump administration is also trying to cut funding for (climate) science, I get the impression that scientists doing science is not what they want. I would love to see how they excuse keeping scientists like me out of the Red Team.

It would also be interesting to see how they will keep the alarmists out. Surely Peter Wadhams would like to defend his position that the Arctic will be ice free this year or the next. Surely Guy McPearson would like to explain why we are doomed and mainstream science, aka science, understates the problem in every imaginable way. I am sure Reddit Collapse of Civilization can suggest many more people with just as much scientific credibility as the people Scott Pruitt would like to invite. I hope they will apply to the Red Team.

That is just one question. Steven Koonin proposes in the Opinion section of the Wall Street Journal that:

A commission would coordinate and moderate the process and then hold hearings to highlight points of agreement and disagreement, as well as steps that might resolve the latter

Does this commission select the topics? Who are these organisers? Who selects them? What are the criteria? After decades of an unproductive blog climate "debate" we already know that there is no evidence that will convince the unreasonable. Will the commission simply write that the Red Team and the Blue Team disagreed about everything? Or will they make an assessment whether it is reasonable for the Red Team to disagree with with evidence?

Clearly Scott Pruitt himself would be the worst possible choice to select the commission. Then the outcome would trivially be: the two teams disagree and Commission Coal Industry declares the Red Team as winner. We already have an NIPCC report with a collection of blog "science". There is no need for a second one.

The then right-wing government of The Netherlands made a similar exercise: Climate Dialogue. They had a somewhat balanced commission and a few interesting debates on, for instance, climate sensitivity, the tropical hotspot, long-term persistence and Arctic sea ice. It was discontinued when it failed to find incriminating evidence. Just like funding for BEST stopped and confirming the general theme of the USA climate "debate": scientists judge studies based on their quality, mitigation "sceptics" based on the outcome.

A somewhat similar initiative in the US was the Climate Change National Forum, where a journalist determined the debating topics by selecting newspaper articles. The homepage is still there, but no longer current. Maybe Pruitt has a few bucks.

---

"This is yet another example of politicians engaging in unhelpful meddling in things they know nothing about."
Ken Caldeira

---

How will Pruitt justify not asking the National Academy of Sciences (NAS), whose job these kind of assessments is, to organise the exercise. Surely the donors of Pruitt will not find the NAS acceptable, they already did an assessment and naturally found the answer that does not fit their economic interests. (Like the findings on climate change of every other scientific organisations from all over the world does not fit their corruption-fuelled profits.)

I guess they will also not ask the Science Division of the White House.

Climate scientist Ken Caldeira called on Scott Pruitt to clarify the hypothesis he wants to test. Given the Trumpian overconfidence, the continual Trumpian own-goals, the Trumpian China-hoax extremism, the Trumpian incompetence and Trump's irrational donors wanting to go after the endangerment finding, I would would not be surprised if they go after the question whether the greenhouse effect exists, whether CO2 is a greenhouse gas or whether the world is warming. Pruitt said he wanted a "discussion about CO2 [carbon dioxide]."

That would be a party. There are many real and difficult questions and sources of uncertainties in climate science (regional changes, changes in extremes, the role of clouds, impacts etc.), but these stupid greenhouse-CO2-warming questions that dominate the low-rated US public "debate" are not among them.

The mitigation sceptical groups are not even able to agree with themselves which of these stupid three questions is the actual problem. I would thus suggest that the climate "sceptics" use their new "scientific method" themselves first to make their chaotic mess of incompatible claims into something.

Red Team PR exercise

Donald Trump has already helped climate action in America enormously by cancelling the voluntary Paris climate agreement. Climate change is slow and global. Everyone hopes someone else will solve it some time and attends to more urgent personal problems. When the climate hoaxer president cancelled the Paris agreement the situation became more dangerous and Americans started paying attention. This surge is seen above in the Google searches for climate change in the USA. This surge was noticeable in Reddit where there was a huge demand for reliable information on climate science and climate action.

The Red Team exercise would give undue weight to a small group of fringe scientists. This is a general problem in America, where many Americans have the impression that extremist positions are still under debate because the fossil fuel industries bought many politicians who in turn say stupid things on cable TV and in opinion sections. These industries also place many ads and in return corporate media is happy to put "experts" on TV that represent their positions. Reality is that 97% of scientists and scientific studies agree that climate change is real and caused by us.

On the optimistic side, just like cancelling Paris made Americans discover that Washington is completely isolated on the world stage in their denial that climate change is a risk, the Red Team exercise could also lead to more American learning how broad the support in the scientific community for climate change is and how strong the evidence.



If the rules of the exercise are clearly unfair, scientists will easily be able to explain why they do not join and ask Pruitt why he thinks he needs such unfair rules. While scientists are generally trusted, the opposite is true for Washington and the big corporations behind Pruitt.

The political donors have set up a deception industry with politicians willing to lie for them, media dedicated to spreading misinformation or at least willing to let their politician deceive the public, they have "think tanks" and their own fake version of the IPCC report and a stable of terrible blogs. These usual suspects writing another piece of misinformation for the EPA will hardly add to the load.

The most tricky thing could be to make clear to the public that science is not resolved in debates. The EPA official E&E News talked to was thinking of a "back-and-forth critique" by government-recruited experts. In science that back and forth is done on paper, to make sure it is clearly formulated, with time to check the claims, read the cited articles and crunch the data. If it is just talk, it is easy to make false claims, which cannot be fact checked on the spot. Unfortunately history has shown that the Red Team will likely be willing to make false claims in public.

If the rules of he exercise are somewhat fair, science will win big time; we have the evidence on our side. At this time, where America pays attention to climate change, that could be a really good advertisement for science and the strength of the evidence that climate change is a huge risk that cannot be ignored.

Concluding, I am optimistic. Either they make the rules unfair. It seems likely they will try to make this exercise into political theatre. Then we can ask them in public why they make the rules so unfair. Don't they have confidence in their position that climate change is a hoax?

If they make the rules somewhat fair, science will win big time. Science will win so much, you will be tired of all the winning, you will be begging, please mister scientist no more winning, I cannot take it any more.

Let me close with John Oliver on Coal. Oliver was sued over this informative and funny piece by coal Barron Robert Murray who also stands behind Scott Pruitt and Trump.

Related reading

Red/Blue & Peer Review by the presidents of the American Geophysical Union (AGU) & the National Academy of Sciences: "Is this a one-off proposal targeting only climate science, or will it be applied to the scientific community’s research on vaccine safety, nuclear waste storage, or any of a number of important policies that should be informed by science?"

Are debatable scientific questions debatable?

Why doesn't Big Oil fund alternative climate research?

My previous post on the Red Cheeks Team.

Great piece by climate scientist Ken Caldeira: Red team, blue team.

Josh Voorhees in Slate: EPA Chief Scott Pruitt Wants to Enlist a “Red Team” to Sow Doubts About Climate Change.

Andrew Freedman in Mashable: EPA to actually hold 'red-team' climate debates, and scientists are livid.

Ars Technica: Playing fossil’s advocate — EPA intends to form “red team” to debate climate science. Agency head reported to desire “back-and-forth critique” of published research by Scott K. Johnson.

The pro-climate libertarian Niskanen Center: Can a Red Team Exercise Exorcise the Climate Debate? May I summarise this optimistic post as: if this new "Red Team" scientific method turns out to be the normal scientific method it would be useful.

Talking Points Memo: Pruitt Is Reportedly Starting An EPA Initiative To Challenge Climate Science.

Audobon's letter to Scott Pruitt: "The oil and gas industry manufactures a debate to avoid legal responsibility for their pollution and to eke out a few more years of profit and power."

Rebecca Leber in Mother Jones (May 2017): Leading Global Warming Deniers Just Told Us What They Want Trump to Do.

Scott Pruitt will likely not ask a court of law. Then they would lose again.

The Red Team method would still be a better scientific method than the authoritarian Soviet method proposed by a comment on a large mitigation sceptical blog, WUWT: Does anyone know if the [American Meteorological Society] gets any federal funding like the National Academy of Science does? ... People sometimes can change their tune when their health of their pocketbook is at stake. Do you really want to get your science from authoritarians abusing the power of the state to determine the truth?

Our wise and climate-cynical bunny thinks the Red Team exercise is a Team B exercise, which is the kind of exercise a Red Team should prevent.

Brad Plumer and Coral Davenport in the New York Times: E.P.A. to Give Dissenters a Voice on Climate, No Matter the Consensus.

Steven Koonin in the Opinion section of Rupert Murdoch's Wall Street Journal (April 2017): A ‘Red Team’ Exercise Would Strengthen Climate Science. (pay-walled)

Kelly Levin of the World Resources Institute: Pruitt’s “Red Team-Blue Team” Exercise a Bad Fit for EPA Climate Science.

Statement by Ken Kimmell, President, Union of Concerned Scientists: EPA to Launch Program Critiquing Climate Science


* Photo at the top of Scott Pruitt at CPAC 2017 by Gage Skidmore under a Creative Commons Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0) license.