Research is at the centre of most business decisions. Businesses use it to understand what their customers want, what works for them, and what doesn’t. Research projects can be lengthy and expensive, so it’s vital that they’re done right. Take New Coke for example. The type of data they collected was insufficient. Coca Cola made their decision on the back of a “sip test”. Respondents enjoyed a sip of New Coke, however after a whole can they realised that the old flavour was superior. Coca Cola didn’t bank on consumer’s familiarity with the old flavour (and nostalgia was strong). Sales plummeted because focus groups collected insufficient data.
Bad research can lead to bad data. But what does that mean? Bad data is anything you see in a dataset that shouldn’t be there. If a large enough proportion of data that’s been collected is “bad”, then it skews the whole sample. A skewed sample means skewed insights. Effective business decisions can’t be made with skewed insights; therefore, a strategy has essentially failed. This can reflect negatively on both the client and the researchers involved.
Understanding exactly where bad data comes from can help improve practice for both researchers and clients. It can create a greater understanding between researcher and client. Having this understanding will make this process much smoother. Clients will understand what they need to provide, researchers will understand what they need to collect.
Bad research and bad data were discussed recently in a panel discussion hosted by Insightflow. Guests Richard Kimber (The Customer Experience Coach) and Ron Sellers (President of Grey Matter Research and Consulting) shared their expertise with us, shedding light on some of the big research dos and don’ts.
Design is one of the most important parts of creating a research project. However, you’re unlikely to get it right first-time. Having friends or colleagues look at your proposed design can be invaluable. Just like general proofreading, outside input can help you identify any weak points before you start your research. If these points aren’t hammered out, you could collect some pretty bad data. It’s worth noting that when it comes to bad research and bad data, you can have one without the other, but it’s often the case that badly designed research is responsible for bad data.
Richard Kimber suggested that designing and writing questionnaires is much more complex than immediately apparent. If an organisation doesn’t have any research professionals, they’ll often create a questionnaire that has no real research rigor. Questions may be leading or closed when they should be open. They might not offer the respondents enough scope to answer the question fully. This can lead to discrepancies in the data.
Another way of looking at it is a great insight from Ron Sellers: “bad data is anything that isn’t telling us the truth”. It may be bad because the source, the collection methods, or the research itself is not good. It could be scales that don’t work or questions that are misleading – or even a questionnaire so boring that people zone out. This is where researchers need to be wary. Making sure these potential problems are avoided might take time that isn’t always available. But collecting bad data can have a negative impact on all of those involved.
So what does bad research design look like?
- Data sources that don’t fit the project
- Collection methods that lack rigor
- Boring or convoluted questions
Issues with data
It’s easy to think that bad data comes from the respondents – it’s them answering the questions after all. Our panelists highlighted other factors that might be in play. Bad research design can not only limit what data you can collect, but it may also influence the responses of your respondents in ways you may not have considered.
According to Ron Sellers, some bad behaviour may not necessarily be the result of a bad respondent. For example, telephone surveys. The respondents could be eager and willing to answer your questions accurately. If they don’t understand the question, they can’t answer it. If they can’t answer the question, researchers can’t collect the right data. There are so many issues surrounding lax research practices, so ensuring that all bases are covered from the get-go are invaluable.
Importantly, ‘bad behaviour’ isn’t just about the responses that are given. Poor quality research design can lead itself to bad behaviour in respondents, says Richard Kimber. “Honesty about the length of your survey is really critical.” If respondents are expecting a short survey and are given 80 questions to answer, they probably won’t answer them all. It may be harder to recruit large samples for longer questionnaires, but it’s more likely that they’ll stick around until the end.
So, what can researchers and clients take away from this?
• Get second, third, or fourth opinions on questionnaires and inventories
• Ensure rigorous analysis of all questions
• Consider the scope of the answers needed from respondents
• Create interesting questionnaires
• Remember that respondents are human – we all make mistakes
By abiding by good research practices, the quality of data will be improved exponentially. Research needs to learn from the mistakes of its forefathers. That way, only great things will happen.