The New Statistics: Effect Sizes, Confidence Intervals, Meta-Analyses
If you have recently read through the requirements of top level journals, such as Nature and Psychological Science, you might have noticed terms like “effect sizes” and “confidence intervals” to accompany, or even supplement, more classical ones like p-values. This is because “classical statistics”, most prominently p-values as an indicator of whether a finding is “significant” (often taken to mean interesting), have recently come under fire.
But what are effect sizes, confidence intervals, and meta-analyses? And (how) are they better than p-values? In this tutorial we will first discuss why p-values have been criticized, including practices that diminish the informativeness of p-values. Then we will go over the logic and interpretation of the new statistics, which are not yet part of standard statistics curricula. For example, we will discuss how effect sizes and meta-analyses can be useful for two main purposes: (1) theory building and evaluation and (2) practical decisions during study design, such as deciding a priori how many participants are necessary to obtain a reliable result.