We could be forgiven for thinking that facts and figures communicate the whole truth and hold the keys to unlocking the value in every future opportunity. New digital tools and technologies not only give us more information about the world around us and the people in it but also help us to know more about ourselves. We can literally monitor every step we take and every calorie we consume. The great hope is that if we can gather enough data, we will have the power to change the things we want to change—and that we can do it without having to face the fear of uncertainty.

Data—that which we can easily measure—is supposed to make us smarter, and maybe it can, but I’d argue that it doesn’t always make us wiser. Many of our actions and reactions can be observed and quantified, but that data doesn’t always expose the truth about why we take or have them. If it did, we would have found a way to stop people smoking cigarettes, overeating, gambling and drinking to excess. All of the health data that scientists use to persuade us to change our behaviour doesn’t necessarily have any effect. Hard facts tell only part of the story.

The Power Of Intuition In A Data-Driven World

Things are no different when it comes to evaluating the potential of ideas. Where was the data that predicted the need for and subsequent success of Google, Facebook and the iPhone, or the decline of Kodak, BlackBerry and orange juice? Which analyst forecast the 250 per cent increase in almond milk sales in the US over the past five years? Who anticipated that yoga pants would unseat jeans in popular culture, to spawn an active-wear revolution that will help the sports-apparel market be worth a predicted $178 billion globally by 2019? And what about colouring books for adults, with an estimated 12 million sold in 2015 in the US alone – who saw that juggernaut coming? When it comes to making predictions about which ideas will fly, we tend to forget that we can only use the information we have at hand about the past or the present to make a judgement call or prediction about the future. We don’t (or can’t) know the significance of things we have no information about, or haven’t yet thought to measure, and can’t possibly know for sure.

And yet we crave certainty, so we keep amassing and putting our faith in data. That faith has been fractured and then shattered by recent political events. According to Steve Lohr and Natasha Singer of The New York Times, all the data (and there was a lot of it) put Hillary Clinton’s chances of winning the 2016 US presidential election at between 70 and 99 per cent. As we know, these forecasts made by experts who had pored over every single possible data point turned out to be far from reliable. Lohr and Singer report ‘a far-reaching change across industries that have increasingly become obsessed with data, the value of it and the potential to mine it for cost-saving and profit-making insights’. However, they also remind us that, ‘data science is a technology advance with trade-offs. It can see things as never before, but also can be a blunt instrument, missing context and nuance.’ This proved to be true in the case of the 2016 presidential election. It was easy to measure how people said they would vote, but far harder to gauge what was in people’s hearts.

Not all of the useful information we can gather can be precisely measured and carefully graphed. What we observe in the everyday about what’s working and what’s not, why this is chosen, and that is rejected, and how the world still turns when people say one thing and do another, can lead to the seemingly insignificant insights that change everything. When we are creating ideas that will exist in the world, we must take that world into account—all of it, not just a logical, thin-sliced or convenient view of it.

We instinctively understand more than we give ourselves credit for and we didn’t learn it all from Britannica, Wikipedia or Google. Every day, we have access to vast amounts of information that we unconsciously collect. While this other kind of data is subjective, it’s still useful, and it can be put to work. If we train ourselves to become more observant, if we pay attention—to our surroundings, to other people, to what’s happening that shouldn’t be, or what’s not happening, that should be—our most mundane experiences can fuel our boldest and most brilliant ideas.

Excerpted from my new book Hunch: Turn Your Everyday Insights Into The Next Big Thing which goes on sale in the US today.

Image by Hernán Piñera.