Tracking online word-of-mouth: The people vs machines debate

Mike Daniels, director of media analysis firm Report International, continues to swear by human analysis even when the content being examined is digital. Metrica founder Mark Westaby used to feel the same, but has come round to automated analysis, and is now one of its most vocal advocates. His new firm Spectrum has just launched its first text mining and sentiment analysis product. Research asked the pair of them to bat the issue back and forth by email.

From:  Mark Westaby
To:       Mike Daniels
Date:   19/5/09 15:22

Dear Mike

The internet is revolutionising the way people air their views. The result is a vast repository of comment and opinion, which has a much more powerful influence on consumers than advertising, marketing and even traditional editorial media coverage.

By monitoring and evaluating the sentiment of online media, including consumer-generated comments, organisations can gain significant advantage. Failing to do so can place them at a dangerous disadvantage. Negative impressions are exacerbated by the highly connected nature of the web, and in a world where sentiment can change and be transmitted to millions at the click of a mouse, evaluation must be virtually instantaneous.

The only way this can be achieved is through highly sophisticated automated systems, as human analysis simply cannot come close to the levels of consistent accuracy or response times required. Fortunately a new generation of technology permits consistently more accurate and cost-effective analysis of sentiment across online as well as traditional media. Critically, this allows monitoring and analysis of sentiment in real time, giving companies the intelligence they require to stay abreast of trends in market perception and the factors driving their reputation.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   25/5/09 19:24

Dear Mark

No one would dispute that the internet has had a profound and irreversible impact on consumers. Digital conversations are taking place in such large quantities that it is all too easy to believe that only automated tools can help us analyse the dynamics of this new word-of-mouth phenomenon.

But there’s an unstated assumption behind the technology promise: that it is necessary to analyse all or a very large percentage of these conversations in case we miss something. Given that the overwhelming majority of blogs and social media sites have an audience of two (the author and his mother), it’s hard to imagine there is much real influence being exerted.

Even if we did want to track every single conversation, your assertion that automated analysis can yield accurate and consistent measures of sentiment flies in the face of research we conducted recently among a global sample of developers, practitioners, academics and users of these tools. We found no system capable of delivering reasonable accuracy levels around sentiment – certainly nowhere near the levels needed for making business decisions.

We have found an enduring demand for human-based measurement programmes – humans can discriminate irony and sarcasm, they can interpret rules, not just follow them, and they are flexible in dealing with new topics and issues… certainly not computers’ strong points.

Cheers
Mike

From:  Mark Westaby
To:       Mike Daniels
Date:   27/5/09 13:24

Dear Mike

Your argument about the blogger with the audience of two misses the point. The internet is a highly connected, non-random network, which means that even bloggers with tiny audiences are just a single click from having huge influence. This is how more and more crises are starting, with what at first appears to be an insignificant issue on a minor blog captured by, say, a journalist using a standard search engine, and that’s why it is necessary to analyse as much internet content as possible, quickly, so that problems can be identified early and nipped in the bud.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   28/5/09 17:46

Dear Mark

Your response illustrates perfectly how proponents of automated analysis always come back to speed as their most significant defining benefit. In crisis situations communicators need tools to help them determine the most appropriate tone and content for their response, as well as identifying where the critical pressure points are and where intervention will be most effective.ommunicators can ill afford to chase down false positives. Every single communicator I speak to about this issue, without exception, demands speed plus direction. Direction about the rate of growth or decline of the crisis issue, and direction about how best to react, and where. Waiting an hour or so more than an automated analysis with its inherent inaccuracies is definitely a price sensible communicators are willing to pay.

Best regards
Mike

From:  Mark Westaby
To:       Mike Daniels
Date:   29/5/09 10:55

Dear Mike

It’s not proponents of automated analysis but changes in our increasingly connected 24/7 world that are determining the value of automated systems for crisis management. If proponents of human analysis really believe that crisis situations make up a minority of work, I suggest they talk to more of the communication professionals for whom every day for the past several months has involved some crisis or another.

You repeat the common criticism that automation is not as accurate as human measurement. This assumes that automated systems are designed to replicate human analysis, which ours are not, for very sound reasons. The brain is a superb piece of machinery for coping with the complexity of human survival, but is actually remarkably poor at the cognitive demands of data coding, which forms the basis of human analysis. Automated systems are far better at this. And as those who support human analysis always fail to point out, analysing irony and sarcasm is one thing, but interpreting and coding them consistently is quite another.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   31/5/09 23:12

Dear Mark

Let me give an example of how easily false positives are generated and how damaging they can be. A client of ours in the technology sector was using an automated tool from another provider to measure sentiment in traditional and online media. One of their goals was to have their brand positively associated with environmental responsibility and green issues, so a great deal of effort was put into building a complex semantic model to measure this. Since the client had carried out no activities around the green brand message, you can imagine the consternation when it showed up strongly in the results feed. It turned out after a fair amount of digging (which, surprise surprise, had to be done by humans) that the word green had been used, correctly, in relation to a green coloured product that was produced with the aid of our client’s equipment. No connection directly to the client’s own products, and certainly no connection with anything to do with the environment. The client received a very rapid – but entirely false – reading of their brand’s media profile.

Of course humans don’t get everything right first time either, but in our experience humans make relatively few errors of judgement, especially in sentiment. Most automated solution providers that I know use human analysts to check the output of their tools, and in some cases to code at least some of the coverage – not the greatest vote of confidence in their automated outputs.

Best regards
Mike

From:  Mark Westaby
To:       Mike Daniels
Date:   3/6/09 15:38

Dear Mike

Your ‘green/environment’ example proves nothing about automated analysis except that there are some companies out there doing it badly – which can be said of human analysis too. There are a number of ways we can avoid false positives pretty much completely, while still delivering powerful sentiment analysis.

It’s actually well established that today’s automated systems can achieve 80% accuracy against humans. But, and it’s a very big but, this assumes an automated system wants to be compared with humans. No amount of quality control or training can disguise the fact that humans are poor at consistently coding large volumes of complex data.

So should an organisation use an automated system for, say, monthly analysis of traditional press cuttings? Probably not, but should a company use human analysis for daily online news and blogs? Probably not. There is a place for both in today’s world. Proponents of human analysis would do well to accept that automated systems, properly used, might actually be rather better than they’re prepared to admit.

Best regards
Mark

From:  Mike Daniels
To:       Mark Westaby
Date:   7/6/09 16:27

Dear Mark

It’s an unpalatable truth for any automated system, but the only genuinely independent determinant of the accuracy of sentiment is human analysis.

In the end, this debate is more about determining where the dividing line lies between automated and human analysis than making a choice of one over the other. I would argue that clients could do much more to guard against quality concerns in media analysis (automated and human) by ensuring market researchers participate in the choice of solution and vendor. If there is one positive thing I would like to see come out of this debate, it is that MR professionals understand the need for their involvement in media analysis decision-making.

All of us in media analysis are still learning to understand the potential and, more importantly, the limitations of automated analysis tools. But as long as humans remain the final point of independent validation, computers can only ever remain a useful support and counting tool.

Cheers
Mike