The dog that didn't bark...interpreting non-significance

Abstract

Hypothesis testing is a crucial component of science. This leads to guidelines (often ignored) in most disciplines including psychology. Unfortunately, most focus on significant effects. Non-significant effects are sidelined, in spite of their importance to scientific progress. This study reports a survey of practicing scientists on how they would report and interpret explicit scenarios with non-significant effects. There was no consensus on interpretation in terms of predicting future results. Respondants agreed about how to report the significance of a hypothesis test. Most chose not to report any descriptive statistics, or the sample size, or anything about power, or sufficient information to enable replication. These results shed light on statistical thinking and so should enable more useable guidelines. For non-significant effects, the importance of a priori power is emphasised.