Rational choice theory could be advanced as a pure set of axioms embodying a formal representation of individual choice under circumstances of uncertainty and strategic interaction. Decision theory incorporates the idea of maximizing utility under circumstances of uncertainty and risk. The basic rule is that the decision-maker could collect information about the utility and probability of each feasible choice, and choose the option that affords the maximum expected utility. (Here the decision-maker is playing against nature.) Game theory expands the range of decision-making situations by giving a representation of strategic interaction: situations in which the actor's outcome depends upon the choices or strategies made by one or more rational opponents. Mathematical game theorists have demonstrated that this problem too admits of rational solution. The actor needs to discover the choices available to him/her and each other player and he/she needs to assign utilities for each possible outcome for each player. It is then demonstrable, for both zero-sum and non-zero-sum games, that there are one or more equilibrium sets of strategies for each player. This means that there is a single strategy or a mixed set of strategies with the property that, given rational choices by the opponent, there is no other strategy available to self that would produce a higher utility for self. (It may be observed that for games with many strategies for each player, discovery of the equilibrium set may not be practical.) Certain two-person games have received a great deal of analysis, including the prisoners' dilemma, the game of chicken, and a range of cooperative games.
So much for the formal theory. In what sense does rational choice theory or game theory provide a basis for an explanation of real social outcomes?
We might begin a response to this question by saying that the axioms have empirical content as descriptions of real human decision-making. Real decision-makers do consider alternative choices in terms of the value and probability of the outcome. Moreover, actors who systematically break the expected utility rule will do less well than those who act according to the rule. So non-expected-utility actors will either learn or disappepear if the stakes are high. Likewise, in strategic situations (those depending on the independent actions of other deliberative agents), actors try to choose their strategy based on an analysis of the future choices of other players. So, once again, rational choice theory appears to capture an important dimension of real human decision making.
As noted in a prior posting, it goes without saying that human reasoners are not entirely conformant to the axioms -- norms intervene, computational limits interfere, and the assesment of risky situations appears to be systematically divergent from the expected-utility model. In fact, these deviations create the subject matter for experimental or behavioral economics. (Daniel Kahneman and Amos Tversky, Judgment under Uncertainty: Heuristics and Biases.)
But given that there is some degree of correspondence between RC axioms and real human reasoning, we might say that RC theory provides an empirically grounded way of modeling certain real situations of decision making and strategic interaction. The pressing question is whether the empirical failures of the axioms as a description of real actors are sufficient to thoroughly invalidate the models. And it would appear that there are specific social settings that are likely to represent something like the pure case of rational decision-making as hypothesized by RC theory.
This implies in turn that there is a realistic basis for treating RC theory as a possible source of real empirically grounded explananations of observed social behavior. For example, suppose we are interested in the distributive features of WTO treaties or the incidence of peasant rebellions in late imperial China. We might model the WTO problem by treating nations as rational actors, attributing a set of utilities and probabilities to them, and using the findings of bargaining theory to predict the nature of the distribution of benefits and burdens in a series of agreements. Or we might regard the occurrence of a rebellion as a choice situation for each potential rebel that involves multiple outcomes, each with a utility and a probability. And we might hypothesize that rebellions will be most frequent in situations where the costs of participation are lowest and the rewards are greatest (James Tong, Disorder Under Heaven: Collective Violence in the Ming Dynasty).
In each case we have taken an abstract mathematical system, used it to create a model of the actual social situation of interest, and have then solved the terms of the model. This solution can then be interpreted as a prediction or post-diction of the actual situation, and we can compare the model's results with empirical and hisistorical facts.
(A very negative assessment of the empirical utility of rational choice theory is offered in Donald Green and Ian Shapiro, Pathologies of Rational Choice Theory: A Critique of Applications in Political Science. Their skepticism is taken on by many authors in Jeffrey Friedman's edited volume, The Rational Choice Controversy: Economic Models of Politics Reconsidered.)