When Statistical Significance Is the Wrong Question

The comfort of a yes/no answer

Statistical significance is appealing because it offers clarity. A result is either significant or it isn’t. Decisions feel easier when framed this way.

Unfortunately, many research questions aren’t suited to binary answers.

Common situations where significance misleads

• Small or moderate sample sizes

• High variability outcomes

• Multiple reasonable analytical choices

• Exploratory or early-stage studies

In these settings, p-values often hide more than they reveal.

Better questions to ask

Instead of asking whether an effect exists, consider: - How large could the effect reasonably be? - How uncertain is the estimate? - Are results stable across assumptions? - What decisions hinge on this result?

Reframing the analysis

Estimation, uncertainty, and sensitivity matter more than thresholds. A non-significant result can still inform decisions, just as a significant one can still be misleading.

Why this distinction matters

When significance is treated as the goal, analyses drift toward overconfidence. When understanding is the goal, uncertainty becomes an asset rather than a weakness.

This way of thinking underpins the approach being taken in the development of InsightSuite.