On the recent warming surge

"Incidentally, when in the journal Science in 2007 we pointed to the exceptionally large warming trend of the preceding 16 years, which was at the upper end of the [climate] model range, nobody cared, because there is no powerful lobby trying to exaggerate global warming."

"And of course in our paper we named natural intrinsic variability as the most likely reason. But when a trend at the lower end of the model range occurs it suddenly became a big issue of public debate, because that was pushed by the fossil fuel climate sceptics’ lobby. There is an interesting double standard there."

Stefan Rahmstorf

Maybe the comment is deleted in shame. At least I cannot find it any more. Someone on Reddit was making exactly the same errors as the fans of the infamous "hiatus" to argue that global warming since 2011 is exploding and we're soon gonna die. Deleted comments can be freely paraphrased.

The data

So let's have a look at what the data actually says. Below are warming curves of two surface temperature datasets and two upper air temperature satellite retrievals. I shortened the warming curve of Berkeley Earth to match the period to the one of GISTEMP. All the other datasets are shown over their full lengths. Taking this step back, looking at the overview, there clearly is no "hiatus" and no "warming surge".

The red dots are the annual temperature anomalies, which are connected by a thin grey line. The long-term trend is plotted by a thick blue line, which is a [[LOESS]] estimate.

If you want to see the "hiatus" you have to think the last two data points away and start in the temperature peak of 1998. Don't worry, I'll wait while you search for it.

A hiatus state of mind

After seeing reality, let's now get into the mindset of a climate "sceptic" claiming to have evidence for a "hiatus", but do this differently by looking at the data since 2011.

Naturally we only plot the recent part of the data, so that context is lost. I am sure the climate "sceptics" do not mind, they also prefer to make their "hiatus" plots start around the huge 1998 El Nino warming peak and not show the fast warming before 1998 for context.




The thick blue lines are quadratic functions fitted to the data. They fit snugly. As you can see both the linear and quadratic coefficients are statistically significant. So clearly the recent warming is very fast, faster than linear and we can soon expect to cross the 2 °C limit, right?

I am sure the climate "sceptics" do not mind what I did. They also cherry picked a period for their "hiatus" and applied a naive statistical test as if they had not cherry picked the period at all. The climate "sceptics" agreeing with Christopher Monckton doing this on WUWT will surely not object to our Redditor doing the same and will conclude with him that the end is nigh.

By the way, I also cherry picked the dataset. The curves of the upper air temperature retrievals are not as smooth and the quadratic terms were not statistically significant. But in case of the "hiatus" debate our "sceptical" friends also only showed data for the datasets showing the least warming. So I am sure they do not object to this now.

If you look at the full period you can see that the variability of the temperature signal is much larger than the variability around the quadratic fit. It is thus clearly a complete coincidence that the curve is so smooth. But, well, the "hiatus" proponents also just look statistically at their cherry picked period and ignore the actual uncertainties (including slowly varying ones.)

Some more knowledgeable catastrophists may worry that it looks as if 2017 may not be another record scorching year. No worries. Also no worries if 2018 is colder again. The Global Warming Policy Foundation thinks it is perfectly acceptable to ignore a few politically inconvenient years and claims that we just have to think 2015 and 2016 away and that thus the death of the "hiatus" has been greatly exaggerated. I kid you not. I have not seen any climate "sceptic" or any "luckwarmer" complaining about this novel statistical analysis method, so surely our Redditor can do the same. Catastrophic warming surge claims are save till at least 2019.

Don't pick cherries

Back to reality. What to do against cherry picking periods? My advice would be: don't cherry pick periods. Not for the global mean temperature, not for the temperature of the Antarctic Peninsula, not for any other climate variable. Just don't do it.

If you have a physical reason to expect a trend change, by all means use that date as start of the period to compute a trend. But for 1998 or 2011 there is no reason to expect a trend change.

Our cheerful Redditor had even a bit more physics in his claim. He said that the Arctic was warming and releasing more carbon dioxide and methane. However, these emissions were not large enough to make the increase in the global greenhouse concentrations speed up. And they count. From our good-natured climate "sceptics" I have never heard a physical explanation why global warming would have stopped in 1998. But maybe I have missed their interest in what happens with the climate system.

If you have no reason to expect a trend change the appropriate test would be for a trend change test at an unknown date. Such a test "knows" that cherry picked periods can have hugely different trends and thus correctly only sees larger trend changes over longer periods as statistically significant. Applied to temperature data the result of such as test does not see any "hiatus" nor any "warming surge".

This test gave the right result during the entire "hiatus" madness period. Don't fool yourself. Use good stats.

Related reading

Stefan Rahmstorf Grant Foster and Niamh Cahill: Global temperature evolution: recent trends and some pitfalls. Environmental Research Letters, 2017. See also "Change points of global temperature".

Cranberry picking short-term temperature trends

Statistically significant trends - Short-term temperature trend are more uncertain than you probably think

How can the pause be both ‘false’ and caused by something?

Atmospheric warming hiatus: The peculiar debate about the 2% of the 2%

My unsolicited advice for a UK government

After my spectacular success as UK election pollster let my try my luck as UK political adviser. Offering outsider continental perspective from a citizen of a countries where coalition governments are normal.

The electoral results could not have been more complicated. The conservatives lost their majority. Against British custom collaboration is necessary.

Wikipedia: United Kingdom general election, 2017

Party	Leader	%	Seats	±
Conservative	Theresa May	42.4%	317	-13
Labour	Jeremy Corbyn	40.0%	262	+33
SNP	Nicola Sturgeon	3.0%	35	-19
Liberal Democrat	Tim Farron	7.4%	12	+3
DUP	Arlene Foster	0.9%	10	+2
Sinn Féin	Gerry Adams	0.8%	7	+3
Plaid Cymru	Leanne Wood	0.5%	4	+1
Green	Jonathan Bartley Caroline Lucas	1.6%	1	0

Conservative Prime Minister Theresa May has proposed to govern together with the [[Democratic Unionist Party]] (DUP), a small deeply conservative Northern Irish protestant party with ties to terrorists after decades of religious terrorism.

That coalition has a majority of only 2 seats.

Other possible Conservative coalitions would be with the Liberal Democrats, with the Scottish National Party and the taboo option with Labour. Reversely, the only realistic coalition for Labour would be with the conservatives.

A Tory-DUP coalition is highly problematic. The DUP will have to get something in return for their collaboration. Typically a small coalition partner needs to get a lot more than their vote share to make it worthwhile and to survive the next election. Given how extreme they are, a majority of only 2 seats will often mean that more moderate conservatives will not be willing to give them what they bargained for and need.

The UK government is, furthermore, supposed to be neutral in Northern Irish problems. I have the impression that the UK does not fully realise that also in the Brexit negotiations Northern Ireland will be an extremely difficult problem. If the UK really wants a hard border this would cut Northern Ireland off from the rest of Ireland, which will cause economic problems for everyone and will make it harder to visit family and friends (for Catholics).

The DUP would at least make sure this problem is not ignored until it is too late, but their role would be far from neutral.


h/t Simon Noone‏

The two seat majority will likely dissipate quite quickly in future special elections. That could lead to another election in the middle of the Brexit negotiations.

The Brexit clock is ticking and the UK will have to make decades worth of political decisions in about one year time. I do not see that happening with a two-seat majority. It would have been a herculean task with a large majority and a capable prime minister.

More logical coalition parties for the Conservatives would be the Liberal Democrats and the Scottish National Party. However, they both do not want to be add-ons to a dominant Conservative party, especially after the bad experiences the Liberal Democrats had in their recent coalition with the Conservatives. Still given the situation I feel these parties should give this option more thought (after the disposal of May).

For the UK it would be highly unorthodox, but I see a coalition of the Conservatives and labour as the best option. Possibly together with the Liberal Democrats and the Scottish National Party. In Germany and The Netherlands we have such coalitions of conservatives and social democrats more often, although only as coalition of last resort. In Switzerland the government is always made up of all parties governing in partnership.

Not being used to coalition governments the UK is lacking a culture of adult behaviour and professional political debate, especially between the Tories and Labour. Maybe a neutral Prime Minister (of the Liberal Democrats) could help to hold a grand coalition together. I realise that that makes the coalition even more unorthodox. But these are also not normal times.

This would be nothing for a non-communicative prime minister with brittle personality and a penchant for kabuki theatre. Also Boris Johnson would also be an utter catastrophe, I hope this does not need explaining, but he may be a useful tool for regicide.

The kind of personality able to lead a grand coalition would also be the best personality for the Brexit negotiations with the EU. Many in the UK seem to see the EU as the enemy who is now laughing at UK's misery. On Twitter someone asked for a German word for the current predicament. Most offered was "Schadenfreude". I would say "Mitleid" (sympathy), "Verunsicherung" (confusion) and "Beunruhigung" (worry).

The UK gutter press like to cite anonymous antagonistic EU assholes. Anyone on twitter will be able to confirm that anonymous assholes are never short in supply. But that a few percent of humanity does not deserve that title does not change geography: Europe and the UK will always be neighbours. People in England seem to often forget that we are even neighbours with a land border.

The EU is interested in friendly relations with its neighbours. A concept that may be difficult to comprehend for (formerly) fascist rags. We are grateful for Monty Python, British expertise on the weather, and for the liberation from fascist occupation. A Russian liberation would have been much less pleasant. Just like young Brits got to know other Europeans as mostly nice people with shared values, European appreciate the UK. One day these young people may want to join their EU friends again.

No deal or a bad deal is damaging for the UK, but thus also unfavourable for the EU. A no-deal Brexit in no way fits to the nearly 50/50 outcome of the referendum.

Thus negotiations based on a common appreciation of long-term friendly relations would give the best results for all. There are so many topics that need to be negotiated that kabuki theatre on Gibraltar or holding EU citizens in the UK hostage are an enormous distraction. Next to EU negotiations the UK will have to negotiate trade relations with dozens of countries and trading blocks. The UK will have to build up institutions on the EU borders and institutions for the safety of drugs, chemical consumer product, banking products, etc.

Before the Brexit referendum we had ludicrous have-your-cake.and-eat-it propaganda. We are now nine months after the referendum and it is about time to have a public discussion what Brexit means. It is unfortunately only a slight exaggeration that this discussion was limited to Brexit is Brexit.  It is not.

It beats me why the UK also wants to leave [[Euratom]], which is not part of the EU. If necessary that could have been done later. They do not only work on nuclear power, but also nuclear medicine. Does the UK want the European Centre for Medium-Range Weather Forecasts, [[ECMWF]], to stay in Reading? It is not officially part of the EU, but has many of the same member states.

Does the UK want to keep student exchanges with the continent, [[Erasmus]]? Does the UK want to stay part of the European research community? If yes how? There are programmes for applied joint European research, [[Horizon 2020]], for fundamental science, [[ERC]], and for scientific coordination, [[COST]]. The UK could pay into those programs. Switzerland pays their researchers directly if they are part of it and in case of COST even gives them some bonus funding for additional research to stimulate participation.

Those are just the examples from my field, science. Similar hard choices will have to be made on so many topics. Personally I would suggest to change as little as possible and (initially) contribute to EU programmes. Just the minimal Brexit program, rights of EU citizens currently in the UK and UK citizens currently in the EU (residence, work, healthcare, benefits, pensions, voting rights), trade and borders (seas, fishing, exploration, Northern Ireland, Gibraltar), are though to get done in about a year. Loosening the ties on all the other topics can be done later.

That requires a friendly and communicative leader and government. Clearly Theresa may should go and Boris Johnson is not an alternative. A grand coalition may be the best way to get a concrete national debate going and to have fruitful negotiations.

[UPDATE. Oh my. When even a reasonable guy like John Oliver buys into the argument that the EU will be extra hard on the UK to scare others from leaving. Because everyone wants to be member of a union that has no upsides, but will beat you up when you leave and diligently works on making it worse to leave yourself.

The EU exists because it has benefits for its members. The UK may not be able to see this after decades of propaganda, but the other EU states surely do. Leaving the EU will mean losing these benefits. No need for additional punishments. Except if you think it is a malicious punishment not to be in applied research program Horizon 2020 when not paying for it or expecting the discounts of a customer loyalty card without having a card.

Otherwise an excellent video with a guest appearance by Lord Buckethead.


]

Related reading

Political support grows for cross-party approach to Brexit negotiations. Theresa May under pressure from across political spectrum to build plural coalition – including Jeremy Corbyn – before EU talks.

Der Spiegel looks at the years ahead: A Wave of Anger Crashes over Britain - Brexit Is Dead.

New York Review of Books: Britain: The End of a Fantasy

Desmog UK: Let's Take A Closer Look at the DUP's Climate Science Denial

Comrade Trend predicts the UK general election outcome

Poll whisperer Nate Silver himself just predicted that in the UK elections the conservatives would win by 7 %. My prediction is 2 %.

The prediction of Silver is a simple average of the last polls of the ten main polling organisations. He uses the same data as I do, kindly gathered by the volunteers of Wikipedia, who also made the plot below.



The statistics normally used to estimate the outcome of elections assumes that there is a fixed value and the surveys are noise around this value. In this case you have to average (in a smart way) to remove the noise of the surveys and get a better estimate of the fixed value.

Wikipedia uses the same assumption to compute the curve of the running average of all polls and takes the average over the 10 most recent polls. The distance between the curves of the Conservatives and Labour in the Wikipedia graph above is about 8 %.

Comrade Trend

However, if the polls are moving it typically keeps on moving in the same direction. Germans call this Genosse Trend, Comrade Trend. If you are going up in the polls Comrade Trend is a great friend.

This could be because it takes time until "everyone" has heard the news, only small part of the population is a news junky. You may also want to wait whether refuting information comes in some time later, whether a campaign changes its position or you may want to talk about the news with your family and friends.

It takes some time until people have found time to read the manifestos (Conservative | Labour). Hopefully many do, there is a real choice this time. Also the strengths and weaknesses of the campaigns mostly stay the same during the campaign. Each time Theresa May openly refuses to answer basic policy questions she loses voters, at least for people like me.

Comrade Trend seems like a good assumption to me. The polls in the UK move so fast that the difference in assumptions really matters this time. The smoothing method [[LOESS]] estimates the trend at a certain time to make the best estimate at that time. It fits a line to the data over a subperiod. Estimating a fixed value would correspond to assuming the line is horizontal over this subperiod. LOESS thus takes Comrade Trend into account. It gives the figure below.



The predictions for election day, this Thursday, were made by assuming a linear trend for the period since the 1st of May.

Election turnout

An important reason that it is hard for polling to predict the outcome of an election is that it is not known who will actually get off the couch and vote. The figure/tweet below shows for one of the polls how much difference various turnout assumptions makes. The spread in the polls is very large this election. That may well be because turnout is very important this time. Older people tend to favour the conservatives and faithfully show up. The question we will not know until election day is how much young people will show up.

For the first time, our results model (using latest poll averages) projects a hung parliament *if* turnout is 78% across all ages. #GE2017 pic.twitter.com/Wv7PzEufLY

— Ian Jones (@ian_a_jones) 2 June 2017

Polling bias

Then there is the question whether polls are not just noisy estimate, but whether they are also biased relative to the election. Nate Silver writes:

Exactly how strong the Conservative tendency to outperform their polls has been depends on where you measure from. Since 1992, Conservatives have beaten their final polling margin over Labour by an average of 4.5 percentage points, and have done so in all but one election. (That was 2010, when both Conservatives and Labour gained ground as Liberal Democrats’ support collapsed, but Labour slightly outperformed its polling margin against the Tories.) Go all the way back to 1945, however, and the average Conservative overperformance is just 1.8 percentage points and is not statistically significant.

I prefer to look at all the data and would say that the bias is not statistically significant. It would be weird for the bias to have become worse, except maybe for really recent elections where parts of the population can no longer be reached with landline telephones. That you can find a period with a higher bias may well be cherry picking. As long as no one can provide a reason for a change in bias since 1992, picking a specific period after looking at the data is statistically suspect.

Outcome

According to Nate Silver UK polling tends to have a relatively high uncertainty and misses the election outcome typically by about 4 %. Estimating a trend is harder than estimating a mean, thus it could be that my 2 % prediction is even a bit more uncertain. Thus if my best estimate is right and the conservatives are only 2 % ahead the election is a toss up: the difference is less than the uncertainty.

Let's see who turns out.

[UPDATE. If you put in my numbers for the conservatives and Labour and add 8 % for the Liberal Democrats, 4 % for UKIP and 3 % for the Greens (estimated from Wikipedia graph), the Electoral Calculus app computes the following seat distribution:

National Prediction: Conservative majority 4

Party	2015 Votes	2015 Seats	Pred Votes	Gains	Losses	Net Change	Pred Seats
CON	37.8%	331	40.0%	6	10	-4	327
LAB	31.2%	232	38.1%	14	1	+13	245
LIB	8.1%	8	7.6%	0	5	-5	3
UKIP	12.9%	1	3.8%	0	1	-1	0
Green	3.8%	1	2.9%	0	0	+0	1
SNP	4.9%	56	4.7%	0	1	-1	55
PlaidC	0.6%	3	0.5%	0	2	-2	1
Minor	0.8%	0	2.5%	0	0	+0	0
N.Ire		18		0	0	+0	18

]

[UPDATE. Just before closing of the election let me publish my last updated graph here. Currently Tories are 4.2 % ahead. That is still within the uncertainty and while it is quite likely that the Conservatives will win, there is still a Trump chance that Labour will win. Let's see who shows up to vote.


]

[POST ELECTION UPDATE. One of the best polls was by YouGov. Good with respect to statistical methodology and accuracy in this election. Interestingly, they did not find a steep trend towards Labour, but had a stable 3% lead for the conservatives since 27 May. Thus Comrade Trend may be more the inertia of the other polling organisations than that of the public.]

Related reading

YouGov, the heavily criticized poll that got it right: UK election: The day after. "But the general picture is clear: the model was a huge success in an election which most politicians, pollsters and commentators got badly wrong."

Nate Silver at 538: Are The U.K. Polls Skewed?

Response to Nate Silver, part one (because it’s early!) This post warns that the averaging method in Nate Silver's post is very basic and not comparable to the advanced methods he uses for US polls.

UK polling report: How the polls have changed since 2015

* The code used to generate my prediction plot is on Github.