P-Value Precision and Reproducibility.

TitleP-Value Precision and Reproducibility.
Publication TypeJournal Article
Year of Publication2011
AuthorsBoos, Dennis D., and Leonard A. Stefanski
JournalAm Stat
Date Published2011

P-values are useful statistical measures of evidence against a null hypothesis. In contrast to other statistical estimates, however, their sample-to-sample variability is usually not considered or estimated, and therefore not fully appreciated. Via a systematic study of log-scale p-value standard errors, bootstrap prediction bounds, and reproducibility probabilities for future replicate p-values, we show that p-values exhibit surprisingly large variability in typical data situations. In addition to providing context to discussions about the failure of statistical results to replicate, our findings shed light on the relative value of exact p-values vis-a-vis approximate p-values, and indicate that the use of *, **, and *** to denote levels .05, .01, and .001 of statistical significance in subject-matter journals is about the right level of precision for reporting p-values when judged by widely accepted rules for rounding statistical estimates.

Alternate JournalAm Stat
Original PublicationP-value precision and reproducibility.
PubMed ID22690019
PubMed Central IDPMC3370685
Grant ListP01 CA142538 / CA / NCI NIH HHS / United States
P01 CA142538-01 / CA / NCI NIH HHS / United States