When the Dashboard Lies: Making Decisions with Data

Every company now describes itself as data-driven, in the same way every company describes itself as innovative — the phrase costs nothing and commits to less. Watch how decisions actually get made, though, and a familiar theatre appears: a leader forms a view, an analyst is dispatched to find supporting numbers, and a slide deck arrives decorated with charts that all point, miraculously, in the direction the leader already chose. That is not data-driven decision-making. It is decision-driven data-making, and the difference between the two determines whether your metrics are instruments or ornaments.

The first discipline is telling vanity metrics from actionable ones. Cumulative registered users is the classic vanity metric: the chart can only ever go up and to the right, it flatters every all-hands meeting, and it cannot inform a single decision. Weekly active users who complete a core action, retention by signup month, revenue per customer — these are actionable, because they can go down, and when they do, they tell you where to look. A useful test for any number on your dashboard: if this metric dropped by a third tomorrow, would we do anything differently? If the honest answer is no, it is decoration.

The second discipline is refusing to let correlation impersonate causation. Your analysis shows that users who enable the mobile app retain twice as well, so someone proposes forcing app installation at signup. But the app did not necessarily cause the retention; your most committed users may simply be the ones who bother to install apps. The only reliable way to separate the two is a controlled experiment — show the feature to a random half of new users and compare cohorts. Where experiments are impossible, at least say the honest sentence aloud: "these move together, and we do not yet know why." That sentence has saved companies millions.

Third, remember Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Reward the support team for closing tickets within an hour, and tickets will close within an hour — resolved or not, because agents learn to close and reopen. Pay sales on signed contracts and revenue will arrive, trailed by churn twelve months later. None of this is dishonesty; it is people rationally optimising what you chose to count. The defence is to pair every target with a guardrail metric that catches the distortion: ticket closure time paired with customer satisfaction, contracts signed paired with second-year renewal.

Fourth, respect the statistics you learned and then forgot. An A/B test with two hundred users per branch will "prove" almost anything if you stare at it long enough. Peeking at results daily and stopping the moment significance appears is the most common way teams manufacture false wins. So is ignoring the novelty effect: any visible change lifts engagement for a week, because users poke at whatever moved. Decide the sample size and the test duration before launch, write down the success threshold, and let the experiment finish. Discipline agreed in advance is the only known cure for wishful reading.

Numbers tell you what is happening; they are strangely silent on why. The retention chart shows users leaving in week two, but it took five customer interviews to learn the actual reason: the export feature they needed was hidden behind an unlabelled icon. This is why mature teams pair quantitative data with qualitative work — session recordings, support transcripts, open-ended interviews. An anecdote is not evidence, but it is an excellent hypothesis machine: the interview suggests the theory, the experiment tests it, the dashboard confirms the fix. Teams that use only one of these instruments are flying with one eye closed.

When you carry data into the boardroom, structure matters as much as substance. Lead with the decision you are asking for, then the two or three numbers that bear on it, then the confidence level and the caveats — in that order. Executives do not need your forty-slide methodology; they need to know what you recommend, how sure you are, and what would change your mind. Above all, resist the temptation to trim the caveats that weaken your case. Presenting the number that hurts your own argument is precisely what makes people trust the numbers that help it; credibility, once spent, does not refresh with the next quarter.

Finally, know when to overrule the dashboard. Data describes the world that already exists; it is structurally conservative. No spreadsheet in 2007 argued for a phone without a keyboard, and no retention metric will justify the bet whose payoff sits three years out. When you enter a new market, ship a category-creating product, or make any call where the feedback loop is longer than your planning cycle, data thins out and judgment must carry the weight. The goal was never to remove human judgment from decisions — it was to stop dressing judgment up as certainty. Be data-informed, brutally honest about which one is speaking, and you will beat both the gut-only romantics and the spreadsheet-only bureaucrats.