 |
 |
Chris Dubreuil Chris is Vice President of Client Development,UK, for Research Now. Chris joined the company in 2006 as Client Development Director, leading the company’s multi-country tracking capabilities and was promoted to Head of Client Development for the UK team in November 2008 and then to VP with the 2009 merger with e-Rewards Market Research. Read the full biography here. |
Survey Design: Back to Basics?
Some ‘bad respondents’ are made, not born...3rd June, 2010
The quality debate dominates the online market research industry, yet much of the focus is on the behaviour of professional or untrustworthy members. But what about our responsibilities to provide respondents with a decent survey experience? Is it possible that we are culpable for driving bad behaviour in surveys by providing poorly designed surveys? And what impact do poorly designed surveys have on panel retention?
As a panel provider we host and provide sample for thousands of surveys every year, and we collect survey satisfaction data from our panel members for each one they complete. Through reviewing this data we have been able to identify the survey design features that drive satisfaction and dissatisfaction, and also the long term impact on panel retention. Our aim is not to point the finger but to promote greater collaboration between clients, researchers and online panel owners for mutual benefit.
The most common questionnaire and survey design feature that causes frustration and dissatisfaction is the inability to give an honest opinion. Panel members are motivated by the opportunity to give their opinions and become frustrated when they are unable to, ‘I was forced to choose an answer when I did not agree with any of the options presented ... there should have been at least an “Other” option’.
Repetition in surveys was also a frequent complaint, particularly on surveys with a loop where the same questions are asked repeatedly for different products or services, ‘Far too many buttons to click. I lost the will to live, and the last part of the survey probably didn't get the attention it deserved’.
Another issue was clarity. Although this can be an issue where translations are required, more common were complaints about the wording of questions and understanding what was required of the respondent. Online surveys lack the assistance of an interviewer to explain questions and therefore clarity of questions and instructions is crucial.
The screening and qualification criteria of the questionnaire design are important both for ensuring good quality data but also a relevant experience for the respondent, ‘Lots of questions about my [confectionery] habits but the survey had failed to identify if I was a [consumer of confectionery] which I am not. This made some of the questions difficult to answer’.
Other respondents complained about surveys that were too in-depth, ‘Unbelievably tiresome and overblown. Do people really feel so strongly about what is essentially over-priced WATER?’ It’s important to remember that respondents do not generally respond and interact with consumer goods and services to the same extent as researchers and brand managers.
These comments also highlight the impact on data quality. Instead of treating panel members with distrust and suspicion we need to consider the complicity of questionnaire and survey design in turning participants ‘bad’. If the questionnaire and survey design force respondents to give misleading or invalid answers there is every chance that they will be identified as a survey offender and could be excluded from the panel.
Not surprisingly surveys that were clear, varied, relevant and user-friendly got positive feedback from panel members. In addition surveys that were thought-provoking, interactive, fun and made the participant feel valued for their contributions further enhanced satisfaction and engagement.
We found that panel members completing a bad survey were three times as likely to drop out as a member completing a good survey. Panel members that are less likely to drop out, and apparently more tolerant and loyal, may be more compliant, thus impacting on the representivity of the data collected. Nonetheless there are few surveys which directly result in subsequent inactivity.
Clearly the factors that determine panel retention are complex and cannot be attributed to survey design alone, however, our recommendations for better questionnaire and survey design are:
| |
1. |
Respect and trust survey participants
Researchers should accept that sometimes it is acceptable for respondents not to have an opinion, not to know much about brands, or not to have much to say. It’s important to balance the desire for rich detail with the acknowledgement that respondents do not generally respond and interact with consumer goods and services to the same extent as researchers, marketers and brand managers.
|
| |
2. |
Back to basics for questionnaire design
The MRS guidelines states that ‘A good questionnaire should engage from the start.’ Questions and tasks should be clear and uncomplicated, avoid repetition, ensure screening questions successfully isolate the sample population, try to make question types as varied as possible and run a pilot.
|
| |
3. |
More collaboration between client, researcher and online panel owner for mutual benefit
The interface between survey and respondent means online questionnaires present greater challenges and opportunities in design than postal, CATI or CAPI surveys. Therefore we propose greater collaboration in questionnaire and survey design to achieve high quality data, and to maximise respondent enjoyment and engagement. This is of mutual benefit to researcher, panel owner, end client and respondent. Researchers should allow their online fieldwork agency to review the questionnaire and propose suggestions. Your online fieldwork provider will be able to advise on tried and tested solutions, for example how to use images, flash, and other rich media to enhance the survey experience and maintain respondent engagement. This in turn will lead to better quality data.
|
| |
4. |
Use new techniques and innovation for continued engagement
Panel owners need to monitor survey satisfaction and be prepared to take action where panel members indicate a negative experience or declining satisfaction in order to preserve their asset.
Panel owners should be aware of new innovations and emerging technologies that keep the survey experience fresh and engaging. Loyal members are frequent survey takers and familiarity can breed contempt, so what looks new and innovative today will look old and tired tomorrow.
|
This study has highlighted the need to change our approach to online panel research, in particular the way that we interact and engage with respondents, but also to return to the principles of good questionnaire design. Online research is often cited as the quick and easy option, but that is a misnomer when it comes to research design. The same, if not more, due care and consideration must be given to questionnaire and survey design to protect the integrity of the data collected and insight realised.
Chris Dubreuil
Comments on this article
Want to share your thoughts...? NOTE: Please note that this board is moderated, and comments are published at the discretion of the site owner. |
Back to 