You Can’t Test Instrument Validity

Instrumental variable (IV) estimation is an important technique in causal inference and applied empirical work. The canonical IV setting looks like the following:

iv

Here, the relationship between X and Y is confounded by unobservable influence factors (denoted by the dashed bidirected arrow). Therefore we cannot estimate the causal effect of X on Y by a simple regression. But since the instrument Z induces variation in X that is unrelated to the unobserved confounders, we can use Z as an auxiliary experiment that allows us to identify the so-called local average treatment effect (or LATE) of X on Y.¹

For this to work it’s crucial that Z doesn’t directly affect Y (i.e., no arrow from Z to Y). Moreover, there shouldn’t be any unobservable confounders (i.e., other dashed bidirected arcs) between Z and Y, otherwise the identification argument breaks down. These two assumptions need to be justified purely based on theoretical reasonings and cannot be tested with the help of data.

Unfortunately, however, you will frequently come across people who don’t accept that the assumption of instrument validity isn’t testable. Usually, these folks then ask you to do one of the following two things in order to convince them:

  1. Show that Z is uncorrelated with Y (conditional on the other control variables in your study), or;
  2. Show that Z is uncorrelated with Y when adjusting for X (again, conditional on the other controls).

Both of these requests are wrong. The first one is particularly moronic. In order to not run into a weak instruments problem we want that Z exerts a strong influence on X. If X also affects Y, there will be a correlation between Z and Y by construction, through the causal chain Z \rightarrow X \rightarrow Y.

The second request is likewise mistaken, because adjusting for X doesn’t d-separate Z and Y. On the contrary, as X is a collider on Z \rightarrow X \dashleftarrow \dashrightarrow Y, conditioning on X opens up the path and thus creates a correlation between Z and Y.²

So both “tests” won’t tell you anything about whether the causal structure in the graph above is correct. Z and Y can be significantly correlated (also condional on X) even though the instrument is perfectly valid. These tests have no discriminating power whatsoever. Instead, all you can do is argue on theoretical grounds that the IV assumptions are fulfilled.

In general, there is no such thing as purely data-driven causal inference. At one point, you will always have to rely on untestable assumptions that need to be substantiated by expert knowledge about the empirical setting at hand. Causal graphs are of great help here though, because they make these assumptions super transparent and tractable. I see way too many people — all across the ranks — who are confused about the untestability of IV assumptions. If we would teach them causal graph methodology more thoroughly, I’m sure this would be less of a problem.

 


¹ Identification of the LATE additionally requires that the effect of Z on X is monotone. If you want to know more about these and other details of IV estimation, you can have a look at my lecture notes on causal inference here.

² I explain the terms d-separation and colliders both here and here (latter source is more technical)

How effective are patents really?

Today, an interesting NBER working paper by Deepak Hegde from NYU Stern and coauthors got published:

We provide evidence on the value of patents to startups by leveraging the random assignment of applications to examiners with different propensities to grant patents. Using unique data on all first-time applications filed at the U.S. Patent Office since 2001, we find that startups that win the patent “lottery” by drawing lenient examiners have, on average, 55% higher employment growth and 80% higher sales growth five years later. Patent winners also pursue more, and higher quality, follow-on innovation. Winning a first patent boosts a startup’s subsequent growth and innovation by facilitating access to funding from VCs, banks, and public investors.

Continue reading How effective are patents really?

Follow-up on “IV regressions without instruments” (technical)

Some time ago I wrote about a paper by Arthur Lewbel in the Journal of Business & Economic Statistics in which he develops a method to do two-stage least squares regressions without actually having an exclusion restrictions in the model. The approach relies on higher moment restrictions in the error matrix and works well for linear or partly linear models. Back then, I expressed concerns that the estimator does not seem to work when an endogenous regressor is binary though; at least not in the simulations I have carried out.

After a bit of email back-and-forth we were able to settle the debate now. Continue reading Follow-up on “IV regressions without instruments” (technical)

IV regressions without instruments (technical)

Arthur Lewbel published a very interesting paper back in 2012 in the Journal of Business & Economic Statistics (ungated version here). The paper attracted quite some attention because it lays out a method to do two-stage least squares regressions (in order to identify causal effects) without the need for an outisde instrumental variable. Continue reading IV regressions without instruments (technical)

The Econ 101 of Agriculture

Wolfram Schenker from Columbia University gave a talk about the impact of climate change on agricultural production at ZEW. The part which struck me the most, I found out later, is part of a paper published in the American Economic Review (Roberts and Schlenker, 2013). Seems like I have a good taste… Continue reading The Econ 101 of Agriculture