Causal inference and random trials

image: Tamil Nadu nutrition study

Nancy Cartwright has spent much of her career probing the assumptions scientists make about causation. She has helped to demonstrate that the Humean assumptions about causation that philosophers (including Carl Hempel) carried into twentieth century philosophy of science don't come close to answering the question correctly, and she has provided many reasons to take seriously the ideas of causal powers and mechanisms rather than governing causal regularities. How the Laws of Physics Lie is an important contribution to the philosophy of science and to realist theory.

Her current book Evidence-Based Policy: A Practical Guide to Doing It Better (with Jeremy Hardie) provides a different critical perspective on causal inference, this time in the context of social policy reasoning. The design and implementation of public policies rest upon a fundamental premise: that we can have evidence-based reasons for predicting what the effects of the policy tool will likely be. But what kind of evidence might that be? The dominant form of evidence favored in the policy science field is random controlled trials: specify the policy intervention P, choose a domain of cases to apply the intervention P to, randomly select cases to receive the intervention (versus the control group that does not), and measure the value of the outcome of interest. If there is a significant difference in the value of the outcome between test group and control group, then we have evidence that P had an effect.

In a nutshell, C&H take issue with the conviction that random controlled trials  (RCT) -- the gold standard of causal inference and experiment in clinical medicine -- provide a basis for expecting that a given policy intervention will have similar effects in the future. Their book can be read as a critique of an excessively statistical understanding of social causality, without realistic analysis of the underlying mechanisms and processes. As Cartwright and Hardie state repeatedly, RCT evidence shows only that the policy worked on the circumstances tested in the study. Instead, they argue that we need to offer evidence about two additional considerations: whether the "causal principle" associated with P will remain the same in new circumstances; and whether the associated conditions necessary for the operation of this principle will be present in the new circumstances.

Here is a fundamental statement of what they mean by a causal principle:

We suppose that causes do not produce their effects by accident, at least not if you are to be able to make reliable predictions about what will happen if you intervene. Rather, if a cause produces an effect, it does so because there is a reliable, systematic connection between the two, a connection that is described in a causal principle. (22)

The statement, "El Nino causes wet winters in North America," is a causal principle. But causal principles are neither universal nor exceptionless:

The fact that causal principles can differ from locale to locale means that you cannot read off that a policy will work here from even very solid evidence that it worked somewhere else. (23)

Here is a more extensive description of this idea:

Causal principles are not universal. They differ from place to place and from time to time. That means that it is not enough for you to know that the policy worked somewhere or even that it has worked at some time here. “It worked there”; it played a positive causal role there. So it was one of the factors from a causal principle that holds there. To predict that it will work here, you need to know that it is one of the factors from a causal principle that holds here. That is what ensures that it can play a positive causal role for you. (50)

Cartwright and Hardie look at causation along the lines of J. L. Mackie's analysis of INUS conditions in The Cement of the Universe: a factor is a cause if it is an "Insufficient but Necessary part of an Unnecessary but Sufficient condition for producing a contribution to the effect" (23). The evidence of favorable CRT studies for a given policy intervention doesn't show that this policy will work in the new circumstances of the new proposed application. In order to draw this inference we need to have confidence that the treatment will play the same causal role in the new setting, and that the necessary conditions will be present in that setting. In other words, we need a more detailed causal analysis of the past and the proposed future.

Here is a sketch of the argument that C&H suggest we need to provide in order to project favorable RCT studies onto future applications:

1. x works there (i.e., x genuinely appears in the causal principle that governs the production of y there post-implementation). 
2. Here and there share that causal principle post-implementation. 
3. The support factors necessary for x to contribute under that principle are present for at least some individuals here post-implementation. 
4. Conclusion. x works here (i.e., x genuinely appears in the causal principle that governs the production of y here post-implementation and the support factors necessary for it to contribute to y are present for at least some individuals here post-implementation). (41)

One way of offering support for premise 2 is to engage in the method of process tracing:

This method confirms the existence of a causal connection between start and finish by confirming, one-by-one, a series of smaller causal steps in between. (38)

Cartwright doesn't put her case in these terms, but I would say that the heart of her intuition is that social outcomes are different from medical outcomes because of their inherent causal heterogeneity. In the social world outcomes like teen pregnancy rates or high school dropout rates are the result of a bundle of conjunctural causal processes. So projecting the results of past random controlled trials into the future requires that we first confirm that the same causal influences and important background conditions are at work. And this is rarely the case. So the fundamental underlying prescription is a pragmatic causal realism about social processes: in order to design and implement policies, we need to have a well developed map of the real causal processes and mechanisms that are underway in the production of the effect we would like to change. In other words, we need to be causal realists if we are to be effective policy makers.

(It is worth observing that this book is deliberately different in tone and specialization from Cartwright's other monographs in the philosophy of causation. The book is designed to be useful for real practitioners of public policy, and it offers clear advice about how to gain the understandings needed in order to validate the idea that a given policy will have desired effects in a novel setting.)