A Take on the "Plausible Instrumental Variables" Debate

In our "quantitative methods in poli sci" seminar today, Andy Gelman and Piero Stanig debated the importance of random assignment of instrumental variables for valid causal inference. The claim being debated was whether it is true that random assignment (literally random---i.e. picking balls from urns, coin flips, etc.) of the instrument in addition to the exclusion restriction on instrumental variables (i.e. no direct effects of the instrument on outcome) and significant first stage are all required to draw valid causal inference from IV regressions. Here's my take on the discussions. If any of you had other interpretations, please comment!

First, it was plain for everyone to see that random assignment of the instrument is not sufficient to ensure that the exclusion restriction is satisfied and therefore is not sufficient to provide leverage for causal inference. As an example, Vietnam draft lottery numbers used as instruments for military service by Angrist, Imbens and Rubin (1993) (gated) were randomly assigned, but the exclusion restriction may have been violated if there were other consequential effects resulting from one's draft lottery number other than military service (e.g. other lifestyle changes that may have resulted from receiving a particular lottery number). Also, it hardly needs to be said that random assignment is not sufficient to ensure that the "significant first stage" assumption holds.

Second, it was understood that random assignment helps to ensure that values of the instrument are ignorable relative to values of the outcome---i.e., values of a randomly assigned instrument are surely not determined by potential outcomes---i.e. values of the instrument are exogneous relative to the outcome (three ways of saying the same thing).

Third, random assignment helps to ensure that the variation in the values of the instrument are not the result of some unobserved factor determining the values of the instrument, the explanatory variable being instrumented, and the outcome. It is in this broader web of relations that random assignment helps to ensure that the exclusion restriction is, at least in part, satisfied. This was probably the most important point that came up during the discussion. This is associated with the "it's culture" argument that is often used to challenge results in comparative politics, with culture being an unobserved factor that simply determines everything. The idea here is that without random assignment, one needs to think hard about the mechanisms through which changes are brought about in the value of the instrumental variable. Is it reasonable to believe that those mechanisms are not sneaking exclusion restriction violations in through the back door? Also, are we confident that a change in the instrument as a result of that mechanism will then produce a change in the explanatory variable in the manner resembling what was estimated in the first stage?

But neither the second or third points above would have us conclude that assignment by way of genuine randomization (picking balls from urns, coin flips, unpredictable weather patterns, etc.) is necessary for the "ignorability", "exclusion restriction", and "significant first stage" assumptions to hold. But nonrandom assignment means that you will have to think harder about whether these requirements are met. I think in the end everyone was willing to accept that.

Aside from the debate, Piero presented an interesting application of some sensitivity analysis methods to test for the consequences of "slight" violations of the exclusion restriction. He pointed us to this working paper by Conley, Hansen and Rossi for more details.

Also, Bob Erikson and Andy proposed using some "common sense" in thinking through whether your IV methods are valid: Suppose you are using rainfall as an instrument for economic growth as a predictor of civil war. Say to yourself, "I just showed that rainfall is associated with civil war." Now think a bit, are you led then to say, "Ah yes, it must be because rainfall determines growth, which we have reason to believe is related to civil war"? Do we believe that "must"?

UPDATE: Andy Gelman has his own thoughts on the debate here.