Statistical Terms in Quantitative Research
- Statistically significant means a result is unlikely due to chance.
- The p-value is the probability of obtaining the difference we saw from a sample (or a larger one) if there really isn’t a difference for all users.
- A conventional (and arbitrary) threshold for declaring statistical significance is a p-value of less than 0.05.
- Statistical significance doesn’t mean practical significance. Only by considering context, we can determine whether a difference is practically significant; that is, whether it requires action.
- The confidence interval around the difference also indicates statistical significance if the interval does not cross zero. It also provides likely boundaries for any improvement to aide in determining if a difference is noteworthy.
- With large sample sizes, you’re virtually certain to see statistically significant results, in such situations it’s important to interpret the size of the difference.
- Small sample sizes often do not yield statistical significance. When they do, the differences tend to be practically significant; that is meaningful enough to warrant action.
