Could You Over-Rely on Analytics?

I'm someone who looks to data and numbers far more often than most people around me. Even for simple daily questions — say, clothing choices — I'm swayed more by large-scale studies that show correlations between a pattern and how it's perceived than by a garment's flashiness. I feel confident making decisions for myself (and advising others) only after I've seen the data speak. After all, when you see an analysis based on real-world data, how could it be misleading? If anything is misleading, it's intuition. Take air travel: many people feel nervous about flying purely for emotional reasons. If they leaned on data, they'd know air travel is in fact the safest mode of transport by far.

Recently, someone planted a question in my head: is it possible to over-rely on analytics or data? At first, the question sounded counterintuitive. Data is definitive, and if it's reliably sourced, it simply reports the truth it observed. But dig a little deeper and you find strong cases where data — or, more precisely, our use of it — has misled people and caused large-scale impact.

Consider eBay. For a long time, internal dashboards showed a tight correlation between the number of digital ads a user saw and the likelihood that user would purchase. Budgets swelled accordingly. Only when the company ran careful holdout tests did the flaw come into focus: many of those ads were being shown to people already on their way to buy — brand searchers and loyal users. Ad exposure and purchase moved together, but the ads weren't causing most of the purchases. Correlation had been mistaken for causation, and expensive decisions followed.

Let's consider another example. A study of the incidence of kidney cancer in the 3,141 counties of the United States reveals a remarkable pattern. The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West.

If you take a moment to analyse these results, your mind might conclude that rural life — clean air and water, fresh food — protects against cancer. But now consider the counties with the highest incidence. They, too, tend to be mostly rural, sparsely populated, and in those same regions. A second, equally tempting story appears: perhaps rural poverty and limited access to care drive higher cancer rates. Both stories can't be true in the way they're framed.

What's really going on is nothing more than the law of small numbers. When sample sizes are small — as they often are in sparsely populated counties — rates bounce around more because of randomness. Small samples produce more extreme values at both ends. The correct lesson is not that rural life prevents cancer or causes it, but that we should be wary of dramatic statistics from small denominators. Weighting by population, reporting uncertainty, or using hierarchical models that "shrink" noisy estimates toward the average tends to wash away the illusion.

None of this means you need advanced statistical training to avoid these fallacies. What it highlights is a deeper issue: our brains are predisposed to find patterns in everything around us — even when none exist. Far too often, analysts and business managers alike fall prey to the mind's tendency to suppress doubt and craft narratives from whatever information is at hand. When we extract "insights" from charts and quantitative summaries, it can feel like rigorous, critical thinking — but it's often the brain's fast, pattern-seeking system drawing associations and spinning stories, while the hard-thinking, sceptical system that questions the numbers stays suppressed.

This tendency has produced enormous losses: militaries seeing patterns where none exist and adopting ineffective strategies; large educational institutions launching sweeping reforms, only to discover they had done the opposite of improving outcomes for students.

Despite this, managers and analysts today are under intense pressure to find "actionable insights," even when they're scarce. As a result, they end up building narratives out of randomness. And because finding patterns in random events enlists the same cognitive machinery a fortune-teller uses to read meaning into a shuffled deck of cards, the hurried insights that result can be about as reliable as a psychic reading — only far more expensive.

What to do about it

Many of these pitfalls can be avoided by being deliberately sceptical of the stories your mind has constructed. Start by framing questions causally: compared with doing nothing — or with the best alternative — what difference does this action actually make? Whenever possible, build a credible counterfactual with randomised experiments, staggered rollouts, or geo-holdouts. When you can't randomise, use quasi-experimental methods and be explicit about the assumptions they require. Measure incrementality rather than simple correlation.

Ground the numbers in plain facts: always show how many cases you're basing a claim on, add a range rather than a single number, and when tiny groups are noisy, "borrow strength" from related groups to smooth the variance. Then stress-test the result — shift the date window, remove a quirky segment, try a different reasonable model. If the effect vanishes under small nudges, it isn't sturdy.

But the spreadsheet alone is never enough. Crosscheck patterns against what users and frontline teams report. Make sure incentives don't reward metric-gaming. Above all, be frank about uncertainty — give a range and the assumptions behind it, not a single magic number. Starting with the caveats is proven to shift the audience's focus away from narrative-weaving and toward critical thinking about the numbers themselves.

Data is one of the most powerful tools we have. The goal isn't to trust it less — it's to use it more honestly.