ARF Results Affirm Online Research Quality

In the US, the Advertising Research Foundation (ARF) has reported findings from its 'Foundations of Quality' (FoQ) project, examining the key issue of quality of online survey response.

The initiative brings together seventeen online panel companies and a number of large buyers of marketing research. The ARF's Online Research Quality Council (ORQC) said a 'sense of urgency' pervaded its meeting earlier this week, resulting in a commitment to a '90-Day plan' which will produce recommendations regarding metrics and business practices, templates, definitions, and training for buyers and sellers of online research.

The study, which analysed over 100,000 completed survey responses from 17 panels in a rigorously designed research project, provided robust answers to a number of the key questions surrounding online data quality.

ARF Chief Research Officer Joel Rubinson said the 'original smoking gun' leading to the investigation was the lack of comparable results across studies. On this, the study found clues as to why switching panels or changing the mix of panels can lead to results that are not comparable across studies: panel practices vary widely, for example in that purchase intent is higher in panels that offer cash incentives. 'There are immediate dividends from FoQ for research buyers and suppliers by understanding that switching panels comes with the risk of discontinuity' says Rubinson.

Other key questions and answers were:

• Is there a small group of 'professional respondents' on everyone's panel, doing it for the money, and gaming the system rather than providing thoughtful answers? - The ARF says the answer is a resounding 'no', adding 'A small proportion of people are on more than one panel, and the panelist pool is not small, in fact comparable or better than mail panels in their heyday. People who are on multiple panels and who take numerous surveys in a month are, on average, better respondents.'
• What drives good survey-taking behavior? - The key to this, the study found, is the length of the survey, with shorter surveys producing fewer 'bad respondents'. In addition, the study sugests that 'The optimal number of surveys taken is higher than most expected, and those who are motivated by wanting to share their opinions rather than being in it for the cash gifts also tend to give more thoughtful, consistent answers.'
• Are people taking the same survey more than once? - Here, the study finds that 'the potential exists but is less than initially reported' and recommends that 'The industry must develop operational approaches to ensure that a survey is not taken more than once.'

Robert Tomei, founder and Co-Chair of the ORQC, comments: 'Now that the facts are in, the ORQC is working in a collaborative manner across research buyers, research agencies and other industry associations to develop a series of recommended approaches, metrics, and procedures that will address many of the quality-related issues that have challenged the industry over the last few years.'

The ORQC's next meeting is for ARF members on September 15th. Founded in 1936, the association is based in New York and is on the web at www.theARF.org .

All articles 2006-23 written and edited by Mel Crowther and/or Nick Thomas, 2024- by Nick Thomas, unless otherwise stated.

Register (free) for Daily Research News

Simple printable version of this article

Email this article to a friend