Experimental methods in sociology

An earlier post noted the increasing importance of experimentation in some areas of economics (link), and posed the question of whether there is a place for experimentation in sociology as well. Here I'd like to examine that question a bit further.

Let's begin by asking the simple question: what is an experiment? An experiment is an intervention through which a scientist seeks to identify the possible effects of a given factor or “treatment”. The effect may be thought to be deterministic (whenever X occurs, Y occurs); or it may be probabilistic (the occurrence of X influences the probability of the occurrence of Y). Plainly, the experimental evaluation of probabilistic causal hypotheses requires repeating the experiment a number of times and evaluating the results statistically; whereas a deterministic causal hypothesis can in principle be refuted by a single trial.

In "The Principles of Experimental Design and Their Application in Sociology" (link) Michelle Jackson and D.R. Cox provide a simple and logical specification of experimentation:

We deal here with investigations in which the effects of a number of alternative conditions or treatments are to be compared. Broadly, the investigation is an experiment if the investigator controls the allocation of treatments to the individuals in the study and the other main features of the work, whereas it is observational if, in particular, the allocation of treatments has already been determined by some process outside the investigator’s control and detailed knowledge. The allocation of treatments to individuals is commonly labeled manipulation in the social science context. (Jackson and Cox 2013: 28)

There are several relevant kinds of causal claims in sociology that might admit of experimental investigation, corresponding to all four causal linkages implied by the model of Coleman’s boat (Foundations of Social Theory)—micro-macro, macro-micro, micro-micro, and macro-macro (link). Sociologists generally pay close attention to the relationships that exist between structures and social actors, extending in both directions. Hypotheses about causation in the social world require testing or other forms of empirical evaluation through the collection of evidence. It is plausible to ask whether the methods associated with experimentation are available to sociology. In many instances, the answer is, yes.

There appear to be three different kinds of experiments that would possibly make sense in sociology.

1. Experiments evaluating hypotheses about features of human motivation and behavior
2. Experiments evaluating hypotheses about the effects of features of the social environment on social behavior
3. Experiments evaluating hypotheses about the effects of “interventions” on the characteristics of an organization or local institution

First, sociological theories generally make use of more or less explicit theories of agents and their behavior. These theories could be evaluated using laboratory-based design for experimental subjects in specified social arrangements, parallel to existing methods in experimental economics. For example, Durkheim, Goffman, Coleman, and Hedström all provide different accounts of the actors who constitute social phenomena. It is feasible to design experiments along the lines of experimental economics to evaluate the behavioral hypotheses advanced by various sociologists.

Second, sociology is often concerned with the effects of social relationships on social behavior—for example, friendships, authority relations, or social networks. It would appear that these effects can be probed through direct experimentation, where the researcher creates artificial social relationships and observes behavior. Matthew Salganik et al’s internet-based experiments (2006, 2009) on “culture markets” fall in this category (Hedström 2006). Hedström describes the research by Salganik, Dodds, and Watts (2006) in these terms:

Salganik et al. (2) circumvent many of these problems [of survey-based methodology] by using experimental rather than observational data. They created a Web-based world where more than 14,000 individuals listened to previously unknown songs, rated them, and freely downloaded them if they so desired. Subjects were randomly assigned to different groups. Individuals in only some groups were informed about how many times others in their group had downloaded each song. The experiment assessed whether this social influence had any effects on the songs the individuals seemed to prefer. 

As expected, the authors found that individuals’ music preferences were altered when they were exposed to information about the preferences of others. Furthermore, and more importantly, they found that the extent of social influence had important consequences for the collective outcomes that emerged. The greater the social influence, the more unequal and unpredictable the collective outcomes became. Popular songs became more popular and unpopular songs became less popular when individuals influenced one another, and it became more difficult to predict which songs were to emerge as the most popular ones the more the individuals influenced one another. (787)

Third, some sociologists are especially interested in the effects of micro-context on individual actors and their behavior. Erving Goffman and Harold Garfinkel offer detailed interpretations of the causal dynamics of social interactions at the micro level, and their work appears to be amenable to experimental treatment. Garfinkel (Studies in Ethnomethodology), in particular, made use of research methods that are especially suggestive of controlled experimental designs.

Fourth, sociologists are interested in macro-causes of individual social action. For example, sociologists would like to understand the effects of ideologies and normative systems on individual actors, and others would like to understand the effects of differences in large social structures on individual social actors. Weber hypothesized that the Protestant ethic caused a certain kind of behavior. Theoretically it should be possible to establish hypotheses about the kind of influence a broad cultural factor is thought to exercise over individual actors, and then design experiments to evaluate those hypotheses. Given the scope and pervasiveness of these kinds of macro-social factors, it is difficult to see how their effects could be assessed within a laboratory context. However, there are a range of other experimental designs that could be used, including quasi-experiments (link) and field experiments and natural experiments (link),  in which the investigator designs appropriate comparative groups of individuals in observably different ideological, normative, or social-structural arrangements and observes the differences that can be discerned at the level of social behavior. Does one set of normative arrangements result in greater altruism? Does a culture of nationalism promote citizens’ propensity for aggression against outsiders? Does greater ethnic homogeneity result in higher willingness to comply with taxation, conscription, and other collective duties?

Finally, sociologists are often interested in macro- to macro-causation. For example, consider the claims that “defeat in war leads to weak state capacity in the subsequent peace” or “economic depression leads to xenophobia”. Of course it is not possible to design an experiment in which “defeat in war” is a treatment; but it is possible to develop quasi-experiments or natural experiments that are designed to evaluate this hypothesis. (This is essentially the logic of Theda Skocpol’s (1979) analysis of the causes of social revolution in States and Social Revolutions: A Comparative Analysis of France, Russia, and China.) Or consider a research question in contentious politics, does widespread crop failure give rise to rebellions? Here again, the direct logic of experimentation is generally not available; but the methods articulated in the fields of quasi-experimentation, natural experiments, and field experiments offer an avenue for research designs that have a great deal in common with experimentation. A researcher could compile a dataset for historical China that records weather, crop failure, crop prices, and incidents of rebellion and protest. This dataset could support a “natural experiment” in which each year is assigned to either “control group” or “intervention group”; the control group consists of years in which crop harvests were normal, while the intervention group would consist of years in which crop harvests are below normal (or below subsistence). The experiment is then a simple one: what is the average incidence of rebellious incident in control years and intervention years?

So it is clear that causal reasoning that is very similar to the logic of experimentation is common throughout many areas of sociology. That said, the zone of sociological theorizing that is amenable to laboratory experimentation under random selection and a controlled environment is largely in the area of theories of social action and behavior: the reasons actor behave as they do, hypotheses about how their choices would differ under varying circumstances, and (with some ingenuity) how changing background social conditions might alter the behavior of actors. Here there are very direct parallels between sociological investigation and the research done by experimental and behavioral economists like Richard Thaler (Misbehaving: The Making of Behavioral Economics). And in this way, sociological experiments have much in common with experimental research in social psychology and other areas of the behavioral sciences.