New publication | Heterogeneity in effect size estimates

Dreber and Johannesson with co-authors provide a framework for studying heterogeneity in effect sizes and the generalizability of empirical findings in the social sciences. Heterogeneity is divided into population heterogeneity, design heterogeneity and analytical heterogeneity. They also estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multi-analyst studies. The results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the design and analytical heterogeneity implies doubling standard errors and confidence intervals to incorporate the added uncertainty. The results imply that the generalizability of individual empirical studies in the social sciences is typically low, and that we need to move towards much larger studies systematically varying populations, research designs and analysis paths; referred to as preregistered prospective meta-analysis.