How do norms influence behavior? More fundamentally, what is a norm?
The question arises for two separate reasons. First, we are interested in knowing why people behave as they do (agency). And second, we are interested in knowing how large social factors (moral and cognitive frameworks, for example) exert influence over individuals (social causation).
The agency question is the more fundamental. Philosophers typically want to answer the question in terms of a model of practical rationality and deliberation. One philosophical answer derives from Aristotle and represents action as the result of rational deliberation. Individuals have a set of goals and values; they have a set of beliefs about the world; and they deliberate about the choices they confront with the aim of achieving their goals consistent with their values, given their beliefs about the world. But philosophers and thinkers as diverse as Nietzsche, Skinner, or Adorno would take issue with this reason-centered theory of action. Other sources of behavior might include unconscious habits or prejudices, instincts, impulses, emotions, and role-playing. A model that incorporates these diverse possible influences on action is unavoidably complex -- but human behavior is likewise complex.
Now let's try to locate the role of norms within a theory of agency. Norms have to do with the reasons and motives that people have for their actions. A norm is a particular kind of influence on action: it is a rule of behavior that leads someone to do something that is otherwise contrary to immediate impulse or interest. Norms get us to do things we don't want to do.
We might say, then, that a norm is a rule of behavior -- for example, "Don't wear shorts to a business meeting," "Don't take coins from the blind man's cup," "Give up your seat on the bus to a disabled person." And a rule can either be internally or externally represented; this means that the the rule may be internalized into the agent's process of decision-making, or it may influence the agent's behavior through punishments and rewards.
Even this simple discussion raises questions, however. Do norms have to be consciously accessible to the agent? Is a moral principle such as "Always keep your promises" a norm, or do ethical principles fall in a different category? Do norms have rational justification, or are they simply an accidental social product like tastes or styles?
As for the ways in which norms influence behavior --
It would seem that there are only a few mechanisms through which norms could possibly influence individual and collective behavior, largely distinguished by being external and internal.
First, it may be that there is an effective mechanism of social education through which each individual develops or activates an internally regulative system of norms or rules. This process can be described as "moral education." The most superficial observation of social behavior indicates that this is so, and social psychologists and sociologists have some ideas about how these systems work. But the bottom line appears fairly clear: individuals who are reared in normal human settings eventually possess action-behavior systems that embody a set of personal norms that influence their conduct. We might draw the analogy to the example of language learning: a normal human child is exposed to the linguistic behavior of others, and arrives at a psychologically realized grammar that guides his/her own language production.
Second, a norm might be embodied in the attitudes, judgment, and behavior of others in such a way that their actions and reactions create incentives and disincentives for the actor. For example, others may possess a set of norms concerning civility in public discourse, and they may punish or reward others according to whether their words are consistent with these norms. In this case the agent conforms to the requirements of the norm out of a calculation of costs and benefits of performance. (It would appear that there is a possibility of circularity here: the externally imposed norm depends upon the internally embodied norm of enforcement of the content of the rule on the part of others.)
Third, it might be the case that there are some norms of inter-personal behavior that are hard-wired. Some norms might have a biological, evolutionary basis. This is the line of thought that sociobiologists have explored with varying levels of success. The emotional responses that adults have to infants and children probably fall in this category -- though it is a conceptually interesting question to consider whether these emotional responses are "norms" or simply features of the affective system. This is relevant to the work that Allan Gibbard does in Wise Choices, Apt Feelings: A Theory of Normative Judgment. Gibbard's fundamental insight seems to be that there must be an evolutionary basis for the "norm-acquisition system" -- the features of human psychology that permit them to acquire certain kinds of moral motives (altruism, friendship, fairness).
So -- what can we say about norms? Human beings act on the basis of deliberation, norm, impulse, and emotion. So our theory of practical rationality and action must make a place for the workings of norms. Second, norms are transmitted to individuals through concrete social processes -- family experiences, schooling, religious institutions, etc. Our theories of social life must incorporate an account of the processes of normative education through which individuals come to possess a particular normative structure. These experiences are the counterpart to the exposure to language on the part of the infant. And third, norms are socially enforced through the actions of others. So norms are socially embodied -- in the institutions of enforcement, the institutions and practices of moral education, and in the practical cognition of the individuals who make up the society.