Specifically, these media organisations claimed, with variations on the wording, that “Some 54 per cent of people [sic] said they would stay in the EU if the 2016 referendum were held tomorrow”. This claim is a false representation of the facts on which it is based.
The facts are as follows:
- Renegade Productions, a company engaged in performing arts and artistic creation and a supplier of content to Channel 4, commissioned a survey from Survation, an online polling company (whose admirable website and downloadable tables we acknowledge as the source of the data below)
- Between 20 October and 2 November 2018, Survation sent questionnaires on a hypothetical second referendum to 20,090 members of its online panel (a group of people who have previously agreed to take part in surveys)
- Of these panel members, 18,789 responded to question Q3, which read as follows: “Imagine there was a referendum tomorrow with the question “Should the United Kingdom remain a member of the European Union or leave the European Union?” How would you vote?”
- Of these respondents, 8092 (43%) said that in that situation they would vote Leave; 9200 (49%) said they would vote Remain: and 1497 (8%) said they were undecided.
- Removing the “undecided” respondents left 17,292 panel members of whom 46% were for Leave and 54% were for Remain.
- Survation concluded (and the media repeated) that “If there was to be another referendum tomorrow on whether the UK should remain a member of the EU or leave, our estimates show that the British public would vote to Remain by 54% to 46%”
So … where’s the fast one?
In simple terms, the fast one is pulled when “the panel” metamorphoses into “the public”.
This switch is practised by practically every polling company that we know of (there is one honourable exception, to which we will return). The organisations that commission the polls, and the media who report the polls, repeat the line.
The reality is different, and follows from the fundamental laws of statistical sampling.
If we want to make a statement about a large group of people, what statisticians call a “universe” – for example, the British electorate – we have no hope of interviewing every one of them. We have to take a sample, and to take the necessary precautions so that the sample represents the “universe”. (The word “represent” has a precise meaning in statistics, and is not the same as “resemble”.) The only way we can do this is to ensure that the sample is random – that every person in the “universe” has an equal chance of being selected. Only then can we make statements about the sample which also apply to the “universe”.
With a true random sample, we can make statements of the following nature:
- Our sample has attribute X (which can be anything that we measured – average income, age, height, weight, favourite football club, political affiliation, or indeed attitude to Brexit)
- Our universe has attribute X, plus or minus Y percent (typically we look for +/-5%), with a high degree of confidence (typically we look for 95%). In other words, we believe that if we repeated the sampling 20 times, in 19 cases out of 20 the attribute would be close to X, within a range of say +/-5%.
In the case in hand, if Survation’s respondents had been selected randomly from the UK electorate, we would have no difficulty saying that the UK electorate was, at the moment of polling and excluding “undecided” voters, 46% for Leave and 54% for Remain. Not only that, but given the size of the electorate (46.8 million people in 2017), and the size of the sample (17,292 excluding undecided), we could add that the results were subject to a margin of error of +/-0.75% with 95% confidence. This margin of error follows from the size of the universe and the size of the sample, when (and only when) the sample is random.
But a random sample, it is not. The respondents were not selected randomly. They were panel members and, by definition, self-selected.
It seems to us significant that neither Survation nor Channel 4 makes any reference to the margin of error. Indeed, with a non-random sample, it is not possible to make any statement about the margin of error.
In short, Channel 4 has compiled and disseminated the views and voting intentions of approximately 20,000 people. The views of the other 46,780,000 persons who make up the UK electorate are unknown, and will remain so until someone does a survey based on a random sample. To the best of our knowledge (here’s our honourable mention), the only research in the UK that is based closely on a random sample is the British Social Attitudes survey (BSA). The 2018-19 BSA survey is underway and is due for reporting in March 2019. Until then, we should treat polls in accordance with what we know of their sampling methodology.
Sources:
https://www.channel4.com/news/major-new-brexit-poll-shows-voters-swinging-towards-remain
https://beta.companieshouse.gov.uk/company/08790830
https://www.survation.com/what-does-the-british-public-now-think-about-brexit/
Dr Bob Edwards is Managing Partner of Edwards Economic Research
