Specification Searching and Significance Inflation Across Time, Methods and Disciplines

Abstract

This paper examines how significance inflation has varied across time, methods and disciplines. Leveraging a unique data set of impact evaluations on 20 kinds of development programmes, I find that results from randomized controlled trials exhibit less significance inflation than results from studies using other methods. Further, randomized controlled trials have exhibited less significance inflation over time, but quasi-experimental studies have not. There is no robust difference between results from researchers affiliated with economics departments and those from researchers affiliated with other predominantly health-related departments. Overall, the biases found appear much smaller than those previously observed in other social sciences. Show more