Published: September 10, 2014
“Statistics should always be treated with a degree of caution.”
Another day, another poll about the Scottish independence referendum. With just a week to go until the historic vote, each morning seems to bring new headlines predicting the outcome. The only problem is, the pollsters don’t always agree: some suggest the Yes campaign is gathering momentum while others assert that the No campaign is still in the lead.
Polls have long been a mainstay of political and wider communications campaigns. They provide valuable insight into how people think, feel and are likely to act. However, as any researcher will tell you, the way in which questions are asked can make a huge difference to the result you get. Statistics should always be treated with a degree of caution.
Major pollsters such as YouGov and Ipsos Mori have approached the Scotland question in the same way, asking “Should Scotland be an independent country?”, but have still seen different results. YouGov’s most recent poll caused shockwaves through Westminster when it put the Yes vote ahead at 51% compared to 49% for the first time. However, these headline figures exclude all those who said they didn’t know (6% of the original sample size), so can these results be trusted? When Ipsos Mori asked the same question last month, not only did the No vote come out on top but 10% of respondents said they didn’t know, suggesting the undecided vote could determine the outcome.
Pollsters such as Panelbase and TNS-BMRB have asked slightly different questions, reminding people that the referendum is coming up and then asking how they intend to vote. Both these have seen an increase in those intending to vote in favour of independence, although again the proportion of undecided voters is high enough to swing the result.
But as Dan Hodges pointed out in the Telegraph, how the question is framed is not the only factor we should consider when analysing the results of a poll; sample size and structure is also critical. While all respected pollsters have focused on those in Scotland, it is not always clear if the respondents are eligible to vote. And surveys carried out on the phone or face-to-face will often get different results to those conducted online, as respondents read into the body language and tone of those asking the questions.
As the saying goes, ‘statistics can be made to prove anything’. Research has shown that the complexity of the question asked, the number of options offered, the inclusion of absurd options and the use of wording that invokes emotion can all significantly impact the way people respond to surveys. So it’s important to read headlines based on polls with a healthy degree of scepticism and interrogate the questions and method behind the results before we allow them to inform our decisions.
The Scottish independence referendum is a once in a lifetime event, and as the vote itself gets closer we can expect to see more and more people trying to predict the outcome. But before you contact your financial adviser or sell your house, remember: 98% of all statistics are made up!
Leave a Reply