Anyone who reads scientific papers has probably come across “p-values“. The Wikipedia entry explains the idea as follows:
In null-hypothesis significance testing, the p-value is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct. A very small p-value means that such an extreme observed outcome would be very unlikely under the null hypothesis. Reporting p-values of statistical tests is common practice in academic publications of many quantitative fields.
These p-values are typically used to evaluate whether the results of some experiment could have happened by chance, or whether there is an alternative explanation – for example, the hypothesis that is being investigated. Wikipedia notes that different p-values can be used, but “by convention”, a p-value cut-off of 0.05 is generally picked. It turns out that there is a rather unexpected reason for that convention, as this paper by Brent Goldfarb and Andrew A. King explained back in 2014 (pointed out by Charles Arthur on Twitter):
We were surprised to learn, in the course of writing this paper, that the p < 0.05 cutoff was established as a competitive response to a disagreement over book royalties between two foundational statisticians. In the early 1920s, Kendall Pearson, whose income depended on the sale of extensive statistical tables, was unwilling to allow Ronald A. Fisher to use them in his new book. To work around this barrier, Fisher created a method of inference based on only two values: p-values of 0.05 and 0.01
Fisher himself later admitted that using a range of p-values was better than his approach based on just two figures – 0.05 and 0.01. But by then, the former of these had become a fossilised part of the scientific method, which researchers used without much thought. In other words, as a result of copyright, and a statistician’s refusal to share some basic calculations, the academic world has been producing research using sub-optimal analytical approaches for nearly a century.
Featured image by Vanderdecken.
