Ask a silly question

“In 20 or 30 years,” says Philip Graves, “we will look back with genuine embarrassment at what we currently do. We’ll look back on this period of market research – where we’ve been seduced by this illusion that we can ask pepole what they think and learn from it – and we’ll try to erase it from history”.

He’s set out his view of the industry in full in his new book, Consumer.ology. Click here to read Brian Tarran’s full interview with Graves. Below, two figures from the industry offer their reactions to the book.

Jeffrey Henning on Consumer.ology

The founder of Vovici says pointing out the shortcomings of surveys and focus groups isn’t enough to kill of market research

?

In Consumer.ology Philip Graves makes the case that belief in market research is as misplaced as belief in numerology or astrology. Businesspeople are afraid to make decisions, and rather than turn to their daily horoscope they conduct surveys to foretell their businesses’ fortunes. All the old tropes condemning research are here: from Henry Ford’s ‘faster horse’, to Steve Job’s “We do no market research”, to at least three different retellings of the research behind New Coke.

The factual errors start on page one, with sentence one: “Market research emerged during the media and advertising boom of the 1950s,” Graves writes, apparently oblivious to the fact that Curtis Publishing created the world’s first formal market research department circa 1911 to help advertisers understand the readers of the Saturday Evening Post. Graves then imaginatively reconstructs the thoughts of these market researchers of the ’50s, before emphasising to the reader that belief in market research can be wrong because belief in other things can be wrong, citing the use of astrology by “Nancy Regan” (sic).

If I were as thoughtless as the author believes consumers are, I would have at this point discarded the book for reasons inexplicable to me. But instead I channelled my inner qualitative researcher and read on, trying to understand how Graves’ world view might be true.

As near as I can construct it, when Graves says “market research” he means “surveys (and focus groups)” and nothing else. In the subtitle of Consumer.ology, “The Market Research Myth, the Truth about Consumers and the Psychology of Shopping,” he is really saying “The Survey Research Myth”. He loves consumer ethnographics, but doesn’t seem to know or use the word ethnography. He also loves ad testing and product trials, but apparently those aren’t ‘market research’ techniques either.

His thesis, in short, is that surveys are almost always bad, focus groups are always bad, and ethnographics, ad tests and product trials are the only truly effective techniques. He digs a shallow grave for survey research by pointing out 13 potential problems with surveys from the standpoint of consumer behaviour, when he could have buried it six feet under if he had referred to some of the research behind the concept of Total Survey Error.

Yes, many surveys are bad. In fact, in some ways, they suffer from more problems than Graves is aware of. And yet, of course, surveys are a technique of proven power, but he dismisses any predictive validity for surveys, even for political polling, despite its long track record of accurately forecasting the behaviour of voters. Graves ignores the important role that surveys play in business-to-business research, where many stakeholders have a say in decisions and where decisions are, if not more rational, certainly less impulsive than in the consumer world. To Graves’ mind, if surveys are ever right they are right by coincidence, like that occasionally accurate horoscope.

Graves falls into the familiar trap of being in love with one or two methodologies rather than treating market research as a toolbox of many different methods, each with its own particular strengths and weaknesses, each with its own role to play. For instance, he sings the praises of behavioural research, especially online, yet researchers will tell you they need surveys and in-depth interviews in order to learn the goals of visitors to such sites. Following what site visitors do is not enough if you can’t figure out what they want to do. True consumer understanding comes from combining the results from multiple research methods, appropriately applied. As Graves points out, consumers find it hard to understand themselves, and it’s even harder for researchers.

If you need to talk a stakeholder out of doing a survey and into doing ethnographic research or product trials, Consumer.ology is the book for you. If you were not aware of the recent science into consumer thinking, buy Predictably Irrational or Freakonomics instead. If you do buy Consumer.ology I look forward to your post-rationalisation of the purchase.

Neil Gains on Consumer.ology

Ditching the old methods is no better than ignoring the new ones, says the founder of Asian research agency Tapestry Works

?

Consumer.ology will be seen as a threat to market research, and rightly so. I am pleased to see a book devoted to recent discoveries in social and medical science including neuroscience, psychology and behavioural economics, and their implications for our industry. Market researchers need to take Graves’ arguments seriously.

But one wonders whether in the process of writing the book, Graves himself has fallen victim to confirmation bias. He compares the research industry to astrology, but fails to offer any scientifically validated alternatives. He favours observation, a vital, and still underused, research tool. However, market research is more than observational research, and it is unrealistic to fully test-market new products as the author suggests. If anything the trend in business is towards earlier, smaller-scale feedback.

Some of the book’s examples are distracting and unhelpful. The New Coke fiasco may have used approaches with serious limitations, but the main lesson is surely that businesses need to ask the right question, focusing on business-based metrics rather than spurious attempts to outsmart Pepsi. The book lacks positive examples of where research has worked, which could have provided deeper and more balanced analysis of the benefits of alternative approaches. For example, despite many flaws, political polling and new product forecasting are often surprisingly accurate, given the complex environments in which they are used to make predictions. And if research is more often wrong than right, the author fails to explain how businesses justify using it.

The book’s key ideas are, however, striking and important. The vast majority of human behaviour is not conscious and rational, and most of the time we are unaware of what we do and why we do it. The implication of this is simple: it is difficult (or impossible) to understand and interpret the vast majority of consumer behaviour through direct questioning. Researchers know this as the ‘say/do’ gap, but perhaps fail to acknowledge just how big that gap is.

When market researchers ask consumers to ‘explain’ their behaviour, they are asking their conscious minds, which had no control or understanding of what happened, but will seek to post-rationalise decisions in a way that allows them to maintain a positive view of themselves.
Direct questions should be avoided, Graves argues, as they are artificial and interfere with our perception. He also makes serious, although familiar, criticisms of group discussions, arguing that our tendency to copy others means they often reach an artificial consensus.

Graves says we must keep in mind that behavioural data is more valid than opinion; that context, question-framing and time frame are all relevant; and that covert research is more valid than overt research.

I believe good researchers already appreciate these issues. My reading of the book reinforces the importance of looking at research problems from multiple perspectives. There is no single approach that avoids all these concerns in a practical and ethical way.

As flawed as many research techniques are, they do offer insights if their limitations are understood and they are integrated with other knowledge. As Graves says, “we are awful witnesses to our own behaviour”. The best remedy for that problem is to rely on several witnesses.

Consumer.ology by Philip Graves is out now, published by Nicholas Brealey Publishing