Does Innovation Policy Really Work?

Last week I had the chance to attend a fantastic conference organized by the Innovation Growth Lab in London on the use of randomized control trials (RCT) in innovation policy evaluation. Field experiments to assess whether certain policies work are already very popular in fields such as development economics. But surprisingly little has been done in OECD countries to this day.

There are many obstacles to overcome in order to establish RCTs in the policy domain. Field experiments involve proper randomization of participants in a treatment and control group. Think of a new educational method that you want to test under real life conditions in the classrooms. Either you will have a hard time explaining to parents in the control class why their children are not allowed to participate. Or the method is perceived to be crap and parents will revolt. Moreover, experimentation requires a government to “admit” that they’re not omniscient—that they have good intentions but are not completely sure how to achieve them. This can be a weapon in the hands of their political enemies. And eventually, field experiments about exciting topics can be simply very expensive. R&D grants for large research projects go in the millions. There are legitimate concerns on the part of policy makers to spend these amounts on a random basis. The conference was mainly organized to convince officials that most of these concerns are exaggerated but also for researchers to share their success stories.

The point that we need better evidence on what we spend our public money on (€152 billion to support businesses across Europe, according to estimates) is also demonstrated impressively by the British What Works Centre for Local Economic Growth. In a recent report they carefully reviewed 1,700 studies that deal with the evaluation of R&D grants, loans, and tax credits and classified them according to the strength of evidence they provide. For this purpose the authors used a variant of the Scientific Maryland Scale (SMS) which you can see in the figure. 

The poorest level of evidence provide simple correlation analyses between a policy measure and an outcome of interest. In level three studies already consider a control group which is adjusted to resemble the treatment group by statistical techniques such as OLS or matching estimators. The gold standard in this classification is of course the randomized control trial. For their review of evidence on R&D grant programs the authors of the What Works Centre only found 42 studies that met their minimum requirements to score at least level three on the SMS. Further, only five studies reached level four. And studies providing the highest level of confidence were absent completely.*

Thus, although papers about R&D subsidies may sometimes appear to be old hat our knowledge about what works in practice is surprisingly little. If you closely inspect the findings in all five studies attaining level four you will also find very mixed results of whether grants have positive effects on firm performance and innovation. So we’re still far from reaching a consensus on the matter. I hope that my recent paper can provide a piece of evidence more to inform the debate.

* To be fair, the report shows a quite strong bias towards so-called experimentalist (or randomista, according to Arthur Lewbel) methods. Theoretical or structural approaches are neglected completely.