Data deluge and insight

It’s great that more data is being brought to us (see the Economist on the data deluge) but in all honesty I am also a little cautious. 

If human behaviour was a physics experiment then I might be keen on the remourseless march of empiricism: the past perfectly predicting the future. But I do not believe this. More data for me simply does not equal better insight.

A nice creative idea, a new way of doing things that previously hadn’t been done before, a ‘Black Swan’ of innovation perhaps, simply takes this apart. I would be cautious of even the notion that statistics is in some ways a fixed science rather than art; to quote Professor Daniel Kahneman, ’The rational model is one in which the beliefs and the desires are supposed to be determined. We were real believers in decision analysis 30 years ago, and now we must admit that decision analysis hasn’t held up.’   

For me data is a startpoint not an endpoint, there is plenty of room for intuition: so don’t lose it.