A few weeks before Hillary Clinton was defeated in the US presidental election I met a guy selling hats emblazoned with both candidates’ names outside the Rockerfeller Center in New York.
‘There will be a big upset in this election. Trump hats are selling like hotcakes,’ he said. It was hard to believe. Just the day before at a behavioural economics conference in Manhattan the academics and experts who had crunched every data point predicted exactly the opposite. The data showed Hillary was on track. But the sales in Trump hats didn’t lie. The data worth paying attention to was closer to home. It was in the stories of the people on the streets of towns where those who wrote the algorithms didn’t live and work.

In our digitally, data-stamped world, facts are king and intuition gets a bad rap. Author Michael Lewis describes the ‘powerful trend to mistrust human intuition and defer to algorithms’ that came about as a result of the work of scientists in the field of behavioural economics. The irony, of course, is that scientists too start out with nothing more than a hunch about what’s worth investigating further. Even those whose job it is to demonstrate proof start out not knowing for sure.

Things are no different when it comes to innovating in the commercial world. Where was the data that predicted the need for and subsequent success of Google, Facebook and the iPhone, or the decline of Kodak, BlackBerry and orange juice? Which analyst forecast the 250 per cent increase in almond milk sales in the US over the past five years? Who anticipated that yoga pants would unseat jeans in popular culture, to spawn an active-wear revolution that will help the sports-apparel market be worth a predicted $178 billion globally by 2019? And what about colouring books for adults, with an estimated 12 million sold in 2015 in the US alone – who saw that juggernaut coming? When it comes to making predictions about which ideas will fly, we tend to forget that we can only use the information we have at hand about the past or the present to make a judgement call or prediction about the future. We don’t (or can’t) know the significance of things we have no information about, or haven’t yet thought to measure, and can’t possibly know for sure. Data may be able to tell us what people do and how they do it, but critically, not why they do it.

Intuition, on the other hand, enables us to tap into our shared human experience to reveal a fundamental truth about what it is people want and need. Often there is no reliable data to go on—which is why the disposable nappy was invented by a frustrated mother, and Warby Parker was the brainchild of a guy who’d gone without glasses for a college semester because he couldn’t afford to replace the ones he’d lost. These stories of curious, empathetic and imaginative people who built successful businesses by seeing problems that were begging for a solution are retold over and over again. Successful entrepreneurs don’t wait for proof that their idea will work. They learn to trust their gut and go.

My new book Hunch: Turn Your Everyday Insights Into The Next Big Thing goes on sale in the UK and Australia today. If you’re in the US, you’ve got just a few more days to wait. Hunch will help you to harness the power of your intuition so you can recognise opportunities others miss and create the breakthrough idea the world is waiting for. Filled with success stories, reflection exercises and writing prompts, I hope it will be your guide to embracing your unique potential and discovering winning ideas.