Communicating science: The “significance” of statistics

 

Statistical significance and scientific importance are distinct, equally valuable aspects of communicating the significance of statistics in scientific research

Communicating the “significance” of statistics in scientific research is often plagued by the fact that the everyday usage of “significant” is very different than the technical meaning of that term. When most people read “significant,” they interpret it to mean “big” or “important.” When statisticians say “significant,” they intend it to mean that the estimated effect size is unlikely to have arisen by chance.

Although both aspects of “significance” are key parts of communicating scientific research, those definitions clearly are not interchangeable! Unfortunately, this confusion between statistical significance and scientific importance has been so widespread (even among scientists!) that statisticians have been making recommendations for years on ways to differentiate between them when communicating scientific results (Amrhein et al., 2019; McShane et al., 2019). Despite those efforts, the two usages continue to be conflated by scientists and laypeople alike.

Also contributing to the misunderstanding is that “statistics” has two different meanings: In everyday use, it refers to numeric facts. In technical usage, it refers to using numeric estimates based on randomly selected samples (subsets of a population) to draw inferences about the population.

What is “statistical significance”?

When statisticians say a result is “statistically significant,” they mean that a statistical test found evidence of an effect based on sample data. The word “statistically” is intended to emphasize that “significant” refers to “unlikely to have arisen by chance” due to how the sample was drawn (Witmer, 2019).

What is “scientific significance?”

The Oxford English Dictionary (2024) defines “significant” as “sufficiently great or important to be worthy of attention; noteworthy; consequential, influential.” What qualifies a statistic to meet those criteria? Even those definitions differ from one another, so let’s parse the distinct meanings before considering how to communicate them effectively.

First, to be “worthy of attention” or “consequential,” a result must be big enough to matter for the topic at hand, such as being clinically or educationally meaningful. A result in the opposite of the expected direction also merits attention.

Second, to be “consequential” or “influential,” a result must be one that can be used to inform a decision or the design of an intervention. To judge whether a study’s results can be applied in those ways, readers need to know several different things about that study, including whether

• The findings can be generalized
• The pattern could be explained by other factors;
• The results can be interpreted as cause-and-effect;
• The presumed cause can be changed.

Thus, the “significance” of statistics in science goes far beyond statistical significance.

Communicating scientific importance

A thorough explanation of a study’s scientific importance will touch on each of the following dimensions, worded to convey the specific topic:

• Convey the size of the effect. If participants in a pilot job training program earned 20% more than similar people who did not undertake the program, that would be a meaningful increase. If they only earned 2% more, we shouldn’t bother replicating that training program.
• Express the direction of the effect. Modify terms like “correlated” or “associated” with words such as “positive” or “inverse” to convey direction. Conveying direction is especially pertinent if the pattern was the opposite of the predicted (or desired!) effect, such as if a new drug actually worsened survival compared to older medications.
• Report the “W’s” (when, where, who) for the study sample, and discuss whether the findings could be generalized to the population or to other places or groups. If a clinical trial of a new medication only studied people ages 25 to 49, we should hesitate to infer that the drug would have the same benefits and side effects among older people.

For more update:

Visit Us 👇

Website Link: https://statisticsaward.com/,

Nomination Link: https://statisticsaward.com/award-nomination/,

Registration Link: https://statisticsaward.com/award-registration/,