Greg Ward Greg Ward is a Director of Fly Research an agency specializing in the latest online and mobile research approaches. He has 25 years experience in the industry particularly with continuous panels. Prior to Fly he worked at YouGov as MD in charge of Brand Index, and at GMI, for whom he set up and managed their European operations. Read the full biography here.

Could do better

online's report card 10 years on

12th September, 2011

A preferred data collection method today by the world’s key research markets, online research has now been around for 10 years. Having promised it would deliver a number of key benefits, a decade on; let’s see what its report card looks like.

School is out, so how has online research performed in the last 10 years?

Out of a score of ten, here is how I would rate its performance.

1. Faster – 9 ½

Turnarounds in online research are much faster than in traditional research. A brief coming in at 4pm can be in the field by 6pm with the results by lunchtime the following day.

2. Ability to gather complex data - 7

Internet research has really improved back office operations so large, complex amounts of data can be processed quickly. But unfortunately, at the front end - the part that respondents see – many online surveys are the same as they were 10 years ago. The internet today is full of colour, movement and interaction, but many online questionnaires are still just electronic versions of old fashioned paper questionnaires.

3. Cheaper - 9 ½

After the initial set up costs, online research is generally much cheaper than other methodologies. And because it is so easy to attach visuals such as product and advertising concepts electronically, printing costs have been eliminated.
Recruiting more representative samples and minority groups are now much more cost effective. Conducting repeat or follow up surveys now is also easier and cheaper.

4. Better quality and insight – 6

Initially perceived as an inferior methodology, the quality of online research has come a long way over the years. Interviewer bias and input errors have been eliminated but this needs to be balanced against the absence of field teams and pilot studies which often pointed out glaring errors before a survey went into field.
Less obviously, online research has changed the base for multi-country studies making it easier to conduct interviews beyond larger countries as well as giving smaller agencies the opportunity to compete in this arena.
Online studies still do not fully reach the elderly and as neither face-to-face nor telephone research can capture ‘cash rich/time poor’ working mothers, or the under 25s, so a mix of methodologies is often necessary.

“Headmaster’s” Comments

Generally, I would say the performance of online research over the past 10 years has been pretty good. The methodology shows promise and can be brilliant on occasion, but I believe the industry must try harder to improve in some areas.

So what and how can we improve?

Response rates

Across the industry most will agree that response rates are down – often on the wrong side of 10% - and these are predicted to decline further. To get better response rates, we need to do a number of things:

1. Design better questionnaires. Well-written questionnaires, designed from an internet-savvy respondent’s point of view will achieve more completes. Unfortunately, many online questionnaires are too long, too complicated and assume too high a level of category involvement. If your survey is too long, drop out will occur and it will be the less engaged – but probably still valuable – customers that go first.

2. Don’t make so many questions compulsory and allow for unanswered questions We can’t force respondents to tell the truth, but if a survey is easier and faster, they are more likely to do so. If you notice the same respondents are leaving out questions on numerous surveys, don’t use them again.

3. Consider why people complete surveys in the first place. For cash? (Sometimes); Out of altruism? (Occasionally). More often, people want to be heard. So it is important for us to show a duty of care by asking good questions.

4. Let respondents do more of the talking, By adding in; ‘so what do you like or not like about’ ….; type questions. Then you can use Wordle.com to create word maps of the results giving clients some incredibly valuable insights.

5. Don’t screen-out halfway through a survey Get all your screeners out of the way as early in the survey as possible.

6. Routinely checking the ‘suspended statistics’ On your surveys is something I would also recommend; in particular the ratio of completes to drop outs. Good surveys work out at a 1000:5, but I have actually seen 1000:1250.

7. Use technological advances to keep your audience interested. Today’s surveys can be colourful and interactive, using sound clips, video as well as click and drag tools. But don’t just use Flash for the sake of it – think about how you could use new technology to improve the way questions are asked.

8. Undertake a detailed registration survey up front. If potential respondents are not prepared to make this effort, they are unlikely to take the time to answer future surveys properly.

9. Continuously recruit new respondents and rigorously weed any ‘dead wood’. Currently, response rates and churn range from 10-40% a year, but the life expectancy of a respondent can double by:

		providing good incentives (preferably cash)
		varying the surveys
		not over surveying (between 6-12 invitations per year)
		communicating regularly

Quality

Concerns over online data quality fall into a number of different areas.

1. Internet Protocol Duplication. Because we use incentives, there is a concern that some ‘professional survey participants’ fill in multiple questionnaires. I believe this only happens about 0.5% of the time. Remember IP addresses are re-used, so be sure a duplicate isn’t a new person before assuming they are answering a survey a second time.

2. Speeding. Again I feel there is undue concern over respondents not paying attention and rushing through surveys. Questions are missed out because they are irrelevant or because of routing. Adding open-ended questions will make it easier to spot time-wasters from more considered respondents.

3. Sample costs. Six years ago in the UK, a typical sample cost was £10 per interview, but now it might be £1.50. Can we realistically expect compliancy with the ESOMAR terms at this price?

4. Consistency checks. Although telling lies and getting away with it is fairly easy on any one survey, doing so across TWO matched surveys 10 days apart is harder because respondents are unlikely to remember what they wrote previously. Here strong agreement with one statement should be matched by strong disagreement with its ‘pair’.

The Future

Online research has evolved to become the most prevalent research methodology in less than 10 years. But we need to keep improving how surveys are presented, recruit better respondents and then work hard to keep them engaged through good communication and a good reward system; while at the same time, removing inactive respondents.

Online response rates can be at least 40% and can increase. Quality and honesty issues can be managed. But the industry needs to recognise that, with the time, effort and costs involved, the price per respondent must come in at a fair price. So clients need to beware. Like with most things in life – you get what you pay for.

Greg Ward

Comments on this article

Want to share your thoughts...?

Want to share your thoughts?

NOTE: Please note that this board is moderated, and comments are published at the discretion of the site owner.