The prediction of Silver is a simple average of the last polls of the ten main polling organisations. He uses the same data as I do, kindly gathered by the volunteers of Wikipedia, who also made the plot below.
The statistics normally used to estimate the outcome of elections assumes that there is a fixed value and the surveys are noise around this value. In this case you have to average (in a smart way) to remove the noise of the surveys and get a better estimate of the fixed value.
Wikipedia uses the same assumption to compute the curve of the running average of all polls and takes the average over the 10 most recent polls. The distance between the curves of the Conservatives and Labour in the Wikipedia graph above is about 8 %.
Comrade Trend
However, if the polls are moving it typically keeps on moving in the same direction. Germans call this Genosse Trend, Comrade Trend. If you are going up in the polls Comrade Trend is a great friend.This could be because it takes time until "everyone" has heard the news, only small part of the population is a news junky. You may also want to wait whether refuting information comes in some time later, whether a campaign changes its position or you may want to talk about the news with your family and friends.
It takes some time until people have found time to read the manifestos (Conservative | Labour). Hopefully many do, there is a real choice this time. Also the strengths and weaknesses of the campaigns mostly stay the same during the campaign. Each time Theresa May openly refuses to answer basic policy questions she loses voters, at least for people like me.
Comrade Trend seems like a good assumption to me. The polls in the UK move so fast that the difference in assumptions really matters this time. The smoothing method [[LOESS]] estimates the trend at a certain time to make the best estimate at that time. It fits a line to the data over a subperiod. Estimating a fixed value would correspond to assuming the line is horizontal over this subperiod. LOESS thus takes Comrade Trend into account. It gives the figure below.
The predictions for election day, this Thursday, were made by assuming a linear trend for the period since the 1st of May.
Election turnout
An important reason that it is hard for polling to predict the outcome of an election is that it is not known who will actually get off the couch and vote. The figure/tweet below shows for one of the polls how much difference various turnout assumptions makes. The spread in the polls is very large this election. That may well be because turnout is very important this time. Older people tend to favour the conservatives and faithfully show up. The question we will not know until election day is how much young people will show up.For the first time, our results model (using latest poll averages) projects a hung parliament *if* turnout is 78% across all ages. #GE2017 pic.twitter.com/Wv7PzEufLY
— Ian Jones (@ian_a_jones) 2 June 2017
Polling bias
Then there is the question whether polls are not just noisy estimate, but whether they are also biased relative to the election. Nate Silver writes:Exactly how strong the Conservative tendency to outperform their polls has been depends on where you measure from. Since 1992, Conservatives have beaten their final polling margin over Labour by an average of 4.5 percentage points, and have done so in all but one election. (That was 2010, when both Conservatives and Labour gained ground as Liberal Democrats’ support collapsed, but Labour slightly outperformed its polling margin against the Tories.) Go all the way back to 1945, however, and the average Conservative overperformance is just 1.8 percentage points and is not statistically significant.I prefer to look at all the data and would say that the bias is not statistically significant. It would be weird for the bias to have become worse, except maybe for really recent elections where parts of the population can no longer be reached with landline telephones. That you can find a period with a higher bias may well be cherry picking. As long as no one can provide a reason for a change in bias since 1992, picking a specific period after looking at the data is statistically suspect.
Outcome
According to Nate Silver UK polling tends to have a relatively high uncertainty and misses the election outcome typically by about 4 %. Estimating a trend is harder than estimating a mean, thus it could be that my 2 % prediction is even a bit more uncertain. Thus if my best estimate is right and the conservatives are only 2 % ahead the election is a toss up: the difference is less than the uncertainty.Let's see who turns out.
[UPDATE. If you put in my numbers for the conservatives and Labour and add 8 % for the Liberal Democrats, 4 % for UKIP and 3 % for the Greens (estimated from Wikipedia graph), the Electoral Calculus app computes the following seat distribution:
National Prediction: Conservative majority 4
| Party | 2015 Votes | 2015 Seats | Pred Votes | Gains | Losses | Net Change | Pred Seats |
|---|---|---|---|---|---|---|---|
| CON | 37.8% | 331 | 40.0% | 6 | 10 | -4 | 327 |
| LAB | 31.2% | 232 | 38.1% | 14 | 1 | +13 | 245 |
| LIB | 8.1% | 8 | 7.6% | 0 | 5 | -5 | 3 |
| UKIP | 12.9% | 1 | 3.8% | 0 | 1 | -1 | 0 |
| Green | 3.8% | 1 | 2.9% | 0 | 0 | +0 | 1 |
| SNP | 4.9% | 56 | 4.7% | 0 | 1 | -1 | 55 |
| PlaidC | 0.6% | 3 | 0.5% | 0 | 2 | -2 | 1 |
| Minor | 0.8% | 0 | 2.5% | 0 | 0 | +0 | 0 |
| N.Ire | 18 | 0 | 0 | +0 | 18 |
[UPDATE. Just before closing of the election let me publish my last updated graph here. Currently Tories are 4.2 % ahead. That is still within the uncertainty and while it is quite likely that the Conservatives will win, there is still a Trump chance that Labour will win. Let's see who shows up to vote.
]
[POST ELECTION UPDATE. One of the best polls was by YouGov. Good with respect to statistical methodology and accuracy in this election. Interestingly, they did not find a steep trend towards Labour, but had a stable 3% lead for the conservatives since 27 May. Thus Comrade Trend may be more the inertia of the other polling organisations than that of the public.]
Related reading
YouGov, the heavily criticized poll that got it right: UK election: The day after. "But the general picture is clear: the model was a huge success in an election which most politicians, pollsters and commentators got badly wrong."Nate Silver at 538: Are The U.K. Polls Skewed?
Response to Nate Silver, part one (because it’s early!) This post warns that the averaging method in Nate Silver's post is very basic and not comparable to the advanced methods he uses for US polls.
UK polling report: How the polls have changed since 2015
* The code used to generate my prediction plot is on Github.
Minor: NS's throwaway "the successful Brexit referendum" is weird. He seems to be unaware that the govt campaigned against it.
ReplyDeleteFair post. And thanks for linking to NS, which was also interesting. For myself, I dunno. My suspicion is that the trend is about played out, but I think my suspicion has near zero value :-).
ReplyDeleteFWIW, the recent "terror attack" is as usual being massively overplayed (contrast almost no-one giving a toss that yet another stupid Yank has shot five people (http://www.bbc.com/news/world-us-canada-40162989)). "Security" usually gives the Tories a boost but they're getting somewhat hammered on funding for police, so who knows?
The trend will not continue forever; that is a source of uncertainty. One of the reasons why polling aggregators try to estimate the mean is that there is a lot of theory on that to estimate the uncertainties. Computing the uncertainty for my method would be complicated, although you can likely do it with a good amount of data.
ReplyDeleteI wrote that the uncertainty of my method is likely larger because the model is more complex. On the other hand time also explains part of the variance, thus the error could actually also be smaller. Someone should write a paper. :-) Or link to an existing one in these comments.
Maybe I should have explicitly have written that I am just having fun, this is just a blog post. I am not a polling expert and their scientific literature may well give reasons why one should not use a LOESS-type model. I mainly found it interesting how important this assumption is and I have never seen such a steep and long rise as the one of Labour in this election.
I think your observation about the trend is interesting, and also that NS doesn't mention such; indeed, his analysis obscures it (although other sites, e.g. the Economist, have clearly shown similar pix to yours). It is also somewhat funny that his post, which is stuffed with as many caveats and maybes as he can think of, nonetheless forgot that one.
ReplyDeleteYes, the graph at the Economist visually looks very similar. However, they find a gap of about 6%. I cannot find a description of how they computed it. I hope I did not make an error.
ReplyDeleteLooking at the polling chart it's easy to see why May called the election when she did.
ReplyDeleteInteresting to see movements between parties. Looks like as soon as the election was called 1/3 of UKIP supporters immediately switched to the Conservatives. Since then a steady movement of Labour-leaning UKIP supporters back to Labour. Also a steady movement of Green and Lib Dem supporters to Labour.
I can't really see much more movement from Greens or Lib Dems to Labour. If remaining UKIP supporters become aware of these polls I think it's possible they could completely collapse, with most going to Conservatives (or not voting). Interesting that the most recent swings seem to be from Conservative directly to Labour though.
One thing about all this: If your estimate is correct I make it that it would be the highest joint Conservative+Labour share since the 1960s. Perhaps ushering in a new strongly 2-party era.
I find it really surprising that the two parties are (together) so large. At least my impression is that both parties are actually very unpopular (that is also why the moves can be so large) or at least the party leaders are unpopular. Plus both parties are pro-Brexit while almost half of the population wanted to remain in the EU.
ReplyDeleteThere's a long-standing general sense that the choice which matters is between Labour and Conservative (in England anyway, a bit different in the other countries) and anything else is tantamount to a protest vote.
ReplyDeleteThe picture in recent years largely stems from what happened in the 2010 election. The Labour+Conservative combined share had been dropping steadily since 1950 when it was typically around 85-95%, with growth/return of the Liberal party then SDP/Liberal Democrats the main factor. In 2010 the combined share hit 65% - the lowest since Labour really became a major party in the 1920s. It seemed like a genuine 3-party system had developed.
What then happened was that the Lib Dems went into a coalition with the Conservatives. The problem was that a large amount of their support was very much anti-Conservative and felt betrayed. In the months after this happened there was a polling swing of 10-15% from the Lib Dems with almost all of it going to Labour, and that's basically stuck ever since. Between 2010 right up until the months prior to the 2015 election Labour consistently polled significantly higher than the Conservatives thanks to this influx.
However, the rise of UKIP was the big factor in 2015. Between 2010 and 2015 their polling increased from 3% to 15%, effectively wiping out the Lib Dem influx to combined Labour+Conservative share. Initially they were a drain on Conservative support but after Cameron's 2013 referendum promise that slightly reversed and Labour support gradually drained towards UKIP. The Greens also took a couple of percent off them.
Since the referendum result there has of course been a question mark over the relevance of UKIP. Their polling numbers actually remained at about 2015 levels right up until the election was announced a couple of months ago. Since then a dramatic drop below 5%, suggesting that UKIP voters feel ready to return to the traditional Labour/Conservative battle (or to not bother voting).
So we're now in a situation where the traditional 3rd party (Liberal Democrats) have been eroded with no sign of return and the 3rd largest party at the previous election suddenly doesn't seem to have a purpose anymore.
Since support for Brexit among voters crosses traditional Conservative/Labour party lines there was really no way either could support remaining in the EU. Labour weren't going to attract Conservative Remain supporters but could very well lose Labour Leave supporters. Being seen to go against Brexit would also very easily turn it into a single-issue election and a single-issue on which they would be demonstrably going against the will of the people - not a good look.
ReplyDeleteEven the Lib Dems - and I can't even imagine what a pro-Brexit Lib Dem voter would look like - haven't gone full anti-Brexit though are proposing what is effectively a second in-out referendum. That doesn't seem to have gained them much traction.
May is a fairly typical Conservative leader in terms of profile and popularity. Corben is atypical and seen as very much an anti-establishment figure in the Bernie Sanders or Trump mold: The degree to which he's taken seriously in the mainstream media is probably out of sync with his popular support.
Thanks very informative. It is also kinda weird that there are (non-regional) third parties of any size in the UK given the electoral system. For the seats in parliament it is more or less the same as the USA, which only has two dominant parties.
ReplyDeleteThere is the psychological difference of a parliamentary system over a presidential system, but theoretically that should not influence your choices for parliament.
If I would be a Brit the promise of a second referendum when it is known what Brexit means would have been a major consideration for me. But it sounds as if Corbyn simply wants to get out.
I would see past Conservative leaders as much more competent. We are now almost a year past the referendum and the Brexit plans of the government have nearly no detail and also otherwise May mostly excels in saying nothing. It is her job to set the main policy lines, but from afar I do not see her doing that.
Hmm. "comrade trend" is very similar to "momentum". However, both are opposed to the other force, which is "reversion to mean". And one might posit a 3rd theory, which would be if there is some kind of new information, that one might expect an asymptotic s-shaped curve moving from an original equilibrium to a new equilibrium as the information diffuses through the population.
ReplyDeleteI'm not sure how one determines which of the 3 is most plausible. The first is unsustainable in the long-term - it might be the short-term version of the s-shaped curve before it flattens. I'm not sure in applying the s-curve how one tells where the new equilibrium will be until the curve has already flattened again, which may make it not useful for predictions.
-MMM
But all three models have in common that the changes are auto-correlated. Which is the LOESS assumption. These three models do not say there is some mean, which is constant and we only do not know its value due to the noise.
ReplyDeleteBut those three reasonable models do mean that it is likely hard to compute the uncertainties of the LOESS model and also to extrapolate it. I only dared to extrapolate it for the UK elections because they were only a few days away. I will not do so yet for the German elections end of September. :-)
Comrade Trend can strut around today. The results currently have the Tories with 42.3% and Labour with 40.1%.
ReplyDelete