New publication | Examining the replicability of online experiments selected by a decision market

Dreber and Johannesson led a large international collaborative project testing the feasibility of using decision markets to select studies for replication and evaluating the replicability of online experiments. A decision market, where social scientists traded on the outcome of replications of 41 systematically selected MTurk social science experiments published in PNAS, were used to select the 12 studies with the lowest and the 12 with the highest market prices for replication (along with two randomly selected studies). The replication rate, defined as a statistically significant effect in the same direction as the original study, was 83% for the top-12 and 33% for the bottom-12 group and 54% overall. The replication effect sizes were on average 45% of the original effect sizes. The observed replicability of MTurk experiments is similar to that of previous systematic replication projects involving laboratory experiments.