Social Europe

  • EU Forward Project
  • YouTube
  • Podcast
  • Books
  • Newsletter
  • Membership

5 Things Everyone Should Know About Opinion Polls

Peter Kellner 21st August 2015

Peter Kellner

Peter Kellner

1) Poll Results Are Snapshots, Not Predictions

Public opinion is seldom fixed. Views change. Good polls tell us what people think at the time they were interviewed. Someone reviewing polling results might make a prediction based on the data, but that is a personal judgement, not a poll finding. Suppose you know that a runner is leading by ten metres in a 1500m race, with one lap to go. You might predict that the runner is likely to win – and you would make your prediction with more confidence knowing the state of the race at that point. But you cannot be certain of the outcome. If you want to know the future, don’t commission a poll. Buy a crystal ball.

2) Good Polls Are Seldom Exactly Right, But Seldom Badly Wrong

Polls obtain responses from a small fraction of the population – typically 1,000-2,000 out of a population of millions. Good polls seek to match their samples to the characteristics of the population as a whole – by age, gender, region, social class etc. But statistical theory warns us that even the best survey is subject to a margin of error.

Suppose a coin is tossed 1,000 times. We would expect it to land heads roughly 500 times, and tails roughly 500 times – but it would be a fluke if it landed EXACTLY 500 times each. Likewise with polls. If 50% of the whole population hold a certain view, a well-conducted poll should produce a finding between 47% and 53%  – but the laws of probability tell us but one time in 20, even a “good” poll will produce a result outside that range. But it’s vanishingly unlikely to be, say, ten points adrift of the truth.

3) A Small, Representative Sample Is Always Better Than A Big, Unrepresentative Sample

Newspapers and television programmes sometimes invite their readers/viewers to text, phone or email their views. They sometimes then say something like: “We now have the verdict of more than 100,000 people  far more than any opinion poll”, and claim that the size of the exercise makes it better than a poll of just 1,000 people.

Nonsense. Here’s a cautionary tale. In the 1936 US elections, a magazine, Literary Digest, elicited the voting intentions of more than two million Americans and said that President Roosevelt would be buried after just one term as President, and that his rival, Alf Landon, would win by a landslide. Never heard of Mr Landon? That’s because he lost badly. Gallup Polls conducted a far smaller, but properly representative, survey – and rightly showed Roosevelt well ahead.

An unrepresentative sample is an unreliable sample – simply making it bigger makes no difference.

4) The Details Matter – Dates, Question-wording, Client

So, you see some polling information. You note that it comes from a reputable company and therefore likely to be properly conducted. But does it mean what it seems to mean? Here are some tests to apply.

When was it conducted? Polls often look at current controversies, at a time when public opinion might be volatile. If the fieldwork for the poll was, say, two weeks ago, it might be a less accurate guide to current public opinion than one conducted two days ago.

Where does the report of the poll findings appear? If it’s a media report, it might tell only part of the story. If it’s put out by a campaigning organisation, it might select only those findings that suit its case. And/or they might – wittingly or unwittingly – abbreviate the questions and/or results in a way that ends up being misleading. To be certain about what questions were asked and what results obtained, the best advice is to go to the polling company’s own website.

Who commissioned the survey? Polling clients often have their own agenda – to promote a cause, a party, a candidate, a product or a point of view. This fact does not necessarily invalidate research they commission. Reputable polling companies make sure that the questions they ask are fair and balanced. But when clients have an agenda, it’s especially important to look under the bonnet and check exactly what questions were asked and what the full results show.

5) Apples And Pears Must Be Compared With Care

Here is a fictional example of real problem. A poll asks people whether they would prefer to spend a sunny summer day in a city or at the seaside. By 60-40%, people say they prefer the seaside. A year later another polling company asks people where would they prefer to spend a sunny summer day: in a city, at the seaside or in the countryside. 45% say the seaside, 30% say the countryside, 25%  say a city. The next day, a new report says that the seaside has slumped in popularity, with the number saying they would like to spend a sunny summer day there, down from 60% to 45%.

It’s obviously nonsense, as the first poll offered only two options while the second offered three. It’s just one example of why it is unwise to compare the findings from different polling companies asking different questions. That example is particularly egregious; sometimes the differences are more subtle – for example, telephone surveys often find different numbers of “don’t knows” than online surveys; as a result the numbers for each of the answer options are liable to be different.

The only safe way to be sure of movements in public opinion is to compare surveys through time by the same polling company, using the same interviewing method (online, phone or face-to-face) and asking the same questions each time. And even then, small differences (say by two or three percentage points) may reflect sampling fluctuations rather than real change.

Peter Kellner

Peter Kellner is a journalist, political commentator and President of the YouGov opinion polling organisation in the United Kingdom.

Harvard University Press Advertisement

Social Europe Ad - Promoting European social policies

We need your help.

Support Social Europe for less than €5 per month and help keep our content freely accessible to everyone. Your support empowers independent publishing and drives the conversations that matter. Thank you very much!

Social Europe Membership

Click here to become a member

Most Recent Articles

09d21a9 The Future of Social Democracy: How the German SPD can Win AgainHenning Meyer
u42198346 How Trump’s Tariff Regime Fuels Global OligarchyGabriel Zucman
u421983462 041df6feef0a 3 Universities Under Siege: A Global Reckoning for Higher EducationManuel Muñiz
u4219836ab582 af42 4743 a271 a4f423d1926d 0 How Trade Unions Can Champion Solidarity in Europe’s Migration DebateNeva Löw
u421983467298feb62884 0 The Weak Strongman: How Trump’s Presidency Emboldens America’s EnemiesTimothy Snyder

Most Popular Articles

u4219834647f 0894ae7ca865 3 Europe’s Businesses Face a Quiet Takeover as US Investors CapitaliseTej Gonza and Timothée Duverger
u4219834674930082ba55 0 Portugal’s Political Earthquake: Centrist Grip Crumbles, Right AscendsEmanuel Ferreira
u421983467e58be8 81f2 4326 80f2 d452cfe9031e 1 “The Universities Are the Enemy”: Why Europe Must Act NowBartosz Rydliński
u42198346761805ea24 2 Trump’s ‘Golden Era’ Fades as European Allies Face Harsh New RealityFerenc Németh and Peter Kreko
startupsgovernment e1744799195663 Governments Are Not StartupsMariana Mazzucato
u421986cbef 2549 4e0c b6c4 b5bb01362b52 0 American SuicideJoschka Fischer
u42198346769d6584 1580 41fe 8c7d 3b9398aa5ec5 1 Why Trump Keeps Winning: The Truth No One AdmitsBo Rothstein
u421983467 a350a084 b098 4970 9834 739dc11b73a5 1 America Is About to Become the Next BrexitJ Bradford DeLong
u4219834676ba1b3a2 b4e1 4c79 960b 6770c60533fa 1 The End of the ‘West’ and Europe’s FutureGuillaume Duval
u421983462e c2ec 4dd2 90a4 b9cfb6856465 1 The Transatlantic Alliance Is Dying—What Comes Next for Europe?Frank Hoffer

S&D Group in the European Parliament advertisement

Cohesion Policy

S&D Position Paper on Cohesion Policy post-2027: a resilient future for European territorial equity”,

Cohesion Policy aims to promote harmonious development and reduce economic, social and territorial disparities between the regions of the Union, and the backwardness of the least favoured regions with a particular focus on rural areas, areas affected by industrial transition and regions suffering from severe and permanent natural or demographic handicaps, such as outermost regions, regions with very low population density, islands, cross-border and mountain regions.

READ THE FULL POSITION PAPER HERE

ETUI advertisement

HESA Magazine Cover

What kind of impact is artificial intelligence (AI) having, or likely to have, on the way we work and the conditions we work under? Discover the latest issue of HesaMag, the ETUI’s health and safety magazine, which considers this question from many angles.

DOWNLOAD HERE

Eurofound advertisement

Ageing workforce
How are minimum wage levels changing in Europe?

In a new Eurofound Talks podcast episode, host Mary McCaughey speaks with Eurofound expert Carlos Vacas Soriano about recent changes to minimum wages in Europe and their implications.

Listeners can delve into the intricacies of Europe's minimum wage dynamics and the driving factors behind these shifts. The conversation also highlights the broader effects of minimum wage changes on income inequality and gender equality.

Listen to the episode for free. Also make sure to subscribe to Eurofound Talks so you don’t miss an episode!

LISTEN NOW

Foundation for European Progressive Studies Advertisement

Spring Issues

The Spring issue of The Progressive Post is out!


Since President Trump’s inauguration, the US – hitherto the cornerstone of Western security – is destabilising the world order it helped to build. The US security umbrella is apparently closing on Europe, Ukraine finds itself less and less protected, and the traditional defender of free trade is now shutting the door to foreign goods, sending stock markets on a rollercoaster. How will the European Union respond to this dramatic landscape change? .


Among this issue’s highlights, we discuss European defence strategies, assess how the US president's recent announcements will impact international trade and explore the risks  and opportunities that algorithms pose for workers.


READ THE MAGAZINE

Hans Böckler Stiftung Advertisement

WSI Report

WSI Minimum Wage Report 2025

The trend towards significant nominal minimum wage increases is continuing this year. In view of falling inflation rates, this translates into a sizeable increase in purchasing power for minimum wage earners in most European countries. The background to this is the implementation of the European Minimum Wage Directive, which has led to a reorientation of minimum wage policy in many countries and is thus boosting the dynamics of minimum wages. Most EU countries are now following the reference values for adequate minimum wages enshrined in the directive, which are 60% of the median wage or 50 % of the average wage. However, for Germany, a structural increase is still necessary to make progress towards an adequate minimum wage.

DOWNLOAD HERE

Social Europe

Our Mission

Team

Article Submission

Advertisements

Membership

Social Europe Archives

Themes Archive

Politics Archive

Economy Archive

Society Archive

Ecology Archive

Miscellaneous

RSS Feed

Legal Disclosure

Privacy Policy

Copyright

Social Europe ISSN 2628-7641