How Replicable Are Statistically Significant Findings?

Speaker: Dr Patrick Vu

Affiliation: University of New South Wales

Location: Room 215, Chamberlain Building (#35), St Lucia Campus

Abstract: This paper studies what conventional significance thresholds imply about the probability that a finding will replicate. To do so, we introduce the posterior replication power curve, which maps a finding's p-value to its expected replication probability within a given empirical literature. We estimate posterior replication curves for experimental economics, psychology, and social science, and validate the approach by showing that it accurately predicts replication outcomes across fields, performing similarly to prediction markets. Our results show that findings which narrowly satisfy conventional significance thresholds have low expected replication probabilities, even in unbiased literatures without publication bias or p-hacking: a p-value of 5% (1%) corresponds to expected replication probabilities ranging from 0.23-0.33 (0.46-0.51) across fields. By contrast, achieving at least an 80% chance of replication requires much smaller p-values thresholds, ranging from 0.00008--0.00035 across fields. Overall, these results suggest that high replicability requires substantially stronger statistical evidence than is implied by conventional significance thresholds.