What Data Can't Tell You About Your Customers

Every product team has a version of the same argument. On one side: the data shows that users who complete onboarding are 3x more likely to retain. On the other: something about the onboarding feels off — the drop-off at step three is hard to explain, and the qualitative feedback from users suggests the whole flow feels like too much too soon. The data side tends to win, because data sounds more objective. The qualitative side is often right.

This isn't an argument against data. It's an argument for understanding what data can and can't do. Analytics tells you what happened — which features were used, which funnels converted, which cohorts retained. It's very good at this. What it struggles to tell you is why any of that happened, what it felt like to be the person on the other end of those numbers, and what would have happened if you'd done something different.

The what vs. the why

Consider churn. A dashboard can tell you the rate at which customers are leaving, which segments are leaving fastest, and approximately when in the customer lifecycle churn tends to happen. What it can't tell you is what those customers were actually experiencing in the weeks before they left — what was frustrating them, what alternatives they were considering, what would have changed their mind.

This is information you can only get by talking to people. Not surveying them — the response rates are too low and the questions too leading — but actually talking to them, listening to what they say and what they don't say, following up on the things that don't quite make sense. This is slower and messier than pulling a report, which is probably why most teams don't do it enough.

The silent majority problem

Data reflects the behaviour of people who stayed — who continued to use the product long enough to generate data. The people who left early, who never converted, who bounced after one session and never came back, are largely invisible in most analytics systems. Their absence from the data doesn't mean their perspective is unimportant — it often means the opposite.

Some of the most important product insights come from understanding why people didn't choose you, didn't finish the onboarding, or signed up and never came back. These insights require actively going out to find those people, which is hard, or designing data collection that captures early-stage behaviour before users disengage, which requires thinking about measurement as part of the product design rather than an afterthought.

Using both

The best product and marketing teams treat data and qualitative research as complementary rather than competing. Data surfaces patterns that deserve investigation. Qualitative research explains those patterns and generates hypotheses that data can test. Neither is sufficient on its own, and the teams that treat one as the answer tend to build good-looking dashboards with questionable decisions downstream.

The discipline is in knowing which question you're trying to answer and which tool is appropriate for answering it — and being honest when the tool you prefer isn't the right one for the job.