In the midst of the current election campaign, many marketers will understandably be viewing the various political marketing campaigns with a keen eye. Already long-term adman John Ansell has been blogging about the parties’ billboard campaigns, and many marketers will undoubtedly be heard exclaiming “I could do better than that” sometime during the next few weeks.
But there is another sector of business also being scrutinised during the campaign – the market researchers conducting the regular political polls. In most countries the main political polls are funded by media channels in order to attract viewers or readers and to provide an impression of ‘serious journalism’. The funds made available for polling are usually inadequate, capitalising on the fact that research companies are usually keen to conduct political polling at minimal, if any profit, for the good brand coverage thus gained.
In New Zealand, the main political polls are commissioned by TVNZ (using Colmar Brunton); The NZ Herald (Digipoll) and TV3 (TNS). As well as its regular formal political surveying TV3 has also been the first to use an online surveying approach via the Buzz Channel for quick-turnaround ‘dipstick’ measures of public opinion. The New Zealand Morgan survey is conducted independently, the survey questions apparently being ‘piggybacked’ onto other regular surveying work that they conduct (not an uncommon practice).
As readers will be aware, the results from these regular polls differ, and will always do so for many perfectly respectable and sound reasons. But the fact that the question ‘why are they different?’ is so frequently puzzled over reveals that these reasons are not well known. As a result, every time conflicting poll results are published the very practice of market research is questioned, and the value of research undermined. What follows is a brief explanation of the variables that lead to this situation, followed by the stark revelation that few market research practitioners are ever heard to mention.
The first key issue to consider is how representative any poll sample is of the wider electorate. Sample quality is typically the result of three factors – the interviewing methodology; the sampling methodology; and the questionnaire subject and length.
Consider the methodology for the classic political poll – the telephone survey into households. Of course this excludes households without landline telephones, typically poor, often rural Maori households. However this non-landline group is increasingly being enlarged by professional households who only use mobile phones and wireless internet, thereby reducing the quality of the traditional landline phone survey.
Yet even if all households were equally contactable, their occupants’ availability to be surveyed varies widely. Good surveyors will always re-contact households at least three times in order to reach those who are frequently elsewhere, and use other screening questions to ensure a good representation of ages, genders, regions and ethnicities. The response rate, and thus sample quality, will even be influenced by the questionnaire duration. Thus the practice of adding the political questions onto an unrelated 15minute interview will yield a poorer sample than if only those polling questions were asked, within a 3-minute interview. The subject matter in itself will also deter many people, people who dislike being asked political questions yet who will still vote.
This election campaign we have also seen the introduction of a new methodology for political surveying in New Zealand, the use of an online panel (the Buzz Channel), used recently for a fast ‘snapshot’ TV3 campaign survey. An online panel is where everyday people are invited to opt-in to a panel (or list) of those who have agreed to be emailed online questionnaires to complete, usually for some form of reward. These panels offer many significant advantages and are an area of significant growth for market research. But their representativeness can vary greatly depending on how they are generated and maintained. A sound panel needs to be active, so that people don’t ‘drop off’ out of disinterest, and although online surveys can be conducted very quickly, only allowing respondents less than a day to respond is akin to interviewing only those people who answer the phone the first time. Thus a political poll surveyed during a narrow window has to be questioned.
In addition to these sampling factors one also needs to consider what questions to ask and how to analyse the responses. For guidance on this matter I spoke to Harm Hartman of Synovate’s Netherlands office, which has produced the most accurate political polling of the last four Dutch elections. The key says Hartman, is to measure and analyse more than one factor, and to use an appropriate, proven algorithm in order to better predict the likely outcome. Such algorithms will be familiar to many markers for the analysis of brand equity and brand switching, and the same can apply to voting habits. For example, Synovate Netherlands will measure a person’s ‘consideration set’ as well as the one party they expect to vote for. They will also account for who was voted for last time (just as marketers measure ‘last purchase’). Together one can gain a much better idea of who a person is likely to actually vote for than if the only question asked was “who would you vote for today”.
Yet whilst one can relatively simply dictate the requirements for a highly accurate political poll, this entire debate rests on the shifting sand that is the human psyche. As opposed to the proven Newtonian principles that engineers can use, market researchers study people, and as such there is no 100% concrete “truth” to uncover. People can answer surveys differently depending on their mood, the ethnicity of the interviewer, the subject matter, ulterior motives or whatever else is happening in their lives at that moment. One piece of gossip heard a moment after being surveyed could change their vote in an instant.
This is not an excuse for market researchers ‘getting things wrong’, but rather, an explanation of why things will never reach 100% accuracy – although speaking as a researcher, it’s certainly fun trying!