Davidson Heath, Matthew Ringgenberg, Merhdad Samadi, and Ingrid Werner recent paper “Reusing
Natural Experiments” might turn into quite an influential paper as they shed doubts on our profession’s habit of reusing natural experiments as identification strategy in several studies. The identification challenge in economics and finance
can be addressed in different ways, one of them being natural experiments, where a certain policy is introduced for some firms, states, households but not others, or is introduced at different points in time, again in a manner that is quasi-random. Given
that the universe of such natural experiments is limited, they are being reused. However, as the authors show, this leads to false positives. As the ultimate null hypothesis being tested in these different paper is the same (what was the effect of the
experiment?), the different tests are not independent of each other. The good news is that there is a solution, developed by Romano and Wolf (2005), in the form of
using adjusted t-statistics to control for the “family-wise error rate”. While the authors focus on two specific natural experiments in finance (enactment of state business laws and regulation SHO), this is obviously a much broader concern.
One of the natural experiments that I have used in a paper with Ross Levine and Alex Levkov is the branch deregulation episode in the US, widely used in many previous papers.
While I am reasonably confident in the robustness of our findings (among other reasons due to the use of different datasets than previous papers in the literature), it certainly makes me think. Maybe an idea for a PhD student to explore this or other
policy-related natural experiments and gauge their robustness. There is also an interesting parallel to the legal origin literature (started by La Porta et al.), which has shown that there are differences between Civil and Common Code countries along
a variety of policy areas (media freedom, labour laws, creditor and minority shareholder rights, business registration etc.). Given that legal origin is related to so many policy areas, it cannot really be used as instrument for any of these. More generally,
it is now widely recognised in the institution-growth literature that it is hard to identify the causal impact of a specific policy on real sector outcomes if you use a natural experiment as instrument, given that a given natural experiment has impact on many