Friday, February 24, 2017

How organizations adapt


Organizations do things; they depend upon the coordinated efforts of numerous individuals; and they exist in environments that affect their ongoing success or failure. Moreover, organizations are to some extent plastic: the practices and rules that make them up can change over time. Sometimes these changes happen as the result of deliberate design choices by individuals inside or outside the organization; so a manager may alter the rules through which decisions are made about hiring new staff in order to improve the quality of work. And sometimes they happen through gradual processes over time that no one is specifically aware of. The question arises, then, whether organizations evolve toward higher functioning based on the signals from the environments in which they live; or on the contrary, whether organizational change is stochastic, without a gradient of change towards more effective functioning? Do changes within an organization add up over time to improved functioning? What kinds of social mechanisms might bring about such an outcome?

One way of addressing this topic is to consider organizations as mid-level social entities that are potentially capable of adaptation and learning. An organization has identifiable internal processes of functioning as well as a delineated boundary of activity. It has a degree of control over its functioning. And it is situated in an environment that signals differential success/failure through a variety of means (profitability, success in gaining adherents, improvement in market share, number of patents issued, ...). So the environment responds favorably or unfavorably, and change occurs.

Is there anything in this specification of the structure, composition, and environmental location of an organization that suggests the possibility or likelihood of adaptation over time in the direction of improvement of some measure of organizational success? Do institutions and organizations get better as a result of their interactions with their environments and their internal structure and actors?

There are a few possible social mechanisms that would support the possibility of adaptation towards higher functioning. One is the fact that purposive agents are involved in maintaining and changing institutional practices. Those agents are capable of perceiving inefficiencies and potential gains from innovation, and are sometimes in a position to introduce appropriate innovations. This is true at various levels within an organization, from the supervisor of a custodial staff to a vice president for marketing to a CEO. If the incentives presented to these agents are aligned with the important needs of the organization, then we can expect that they will introduce innovations that enhance functioning. So one mechanism through which we might expect that organizations will get better over time is the fact that some agents within an organization have the knowledge and power necessary to enact changes that will improve performance, and they sometimes have an interest in doing so. In other words, there is a degree of intelligent intentionality within an organization that might work in favor of enhancement.

This line of thought should not be over-emphasized, however, because there are competing forces and interests within most organizations. Previous posts have focused on current organizational theory based on the idea of a "strategic action field" of insiders and outsiders who determine the activities of the organization (Fligstein and McAdam, Crozier; link, link). This framework suggests that the structure and functioning of an organization is not wholly determined by a single intelligent actor ("the founder"), but is rather the temporally extended result of interactions among actors in the pursuit of diverse aims. This heterogeneity of purposive actions by actors within an institution means that the direction of change is indeterminate; it is possible that the coalitions that form will bring about positive change, but the reverse is possible as well.

And in fact, many authors and participants have pointed out that it is often enough not the case that the agents' interests are aligned with the priorities and needs of the organization. Jack Knight offers persuasive critique of the idea that organizations and institutions tend to increase in their ability to provide collective benefits in Institutions and Social Conflict. CEOs who have a financial interest in a rapid stock price increase may take steps that worsen functioning for shortterm market gain; supervisors may avoid work-flow innovations because they don't want the headache of an extended change process; vice presidents may deny information to other divisions in order to enhance appreciation of the efforts of their own division. Here is a short description from Knight's book of the way that institutional adjustment occurs as a result of conflict among players of unequal powers:
Individual bargaining is resolved by the commitments of those who enjoy a relative advantage in substantive resources. Through a series of interactions with various members of the group, actors with similar resources establish a pattern of successful action in a particular type of interaction. As others recognize that they are interacting with one of the actors who possess these resources, they adjust their strategies to achieve their best outcome given the anticipated commitments of others. Over time rational actors continue to adjust their strategies until an equilibrium is reached. As this becomes recognized as the socially expected combination of equilibrium strategies, a self-enforcing social institution is established. (Knight, 143)
A very different possible mechanism is unit selection, where more successful innovations or firms survive and less successful innovations and firms fail. This is the premise of the evolutionary theory of the firm (Nelson and Winter, An Evolutionary Theory of Economic Change). In a competitive market, firms with low internal efficiency will have a difficult time competing on price with more efficient firms; so these low-efficiency firms will go out of business occasionally. Here the question of "units of selection" arises: is it firms over which selection operates, or is it lower-level innovations that are the object of selection?

Geoffrey Hodgson provides a thoughtful review of this set of theories here, part of what he calls "competence-based theories of the firm". Here is Hodgson's diagram of the relationships that exist among several different approaches to study of the firm.



The market mechanism does not work very well as a selection mechanism for some important categories of organizations -- government agencies, legislative systems, or non-profit organizations. This is so, because the criterion of selection is "profitability / efficiency within a competitive market"; and government and non-profit organizations are not importantly subject to the workings of a market.

In short, the answer to the fundamental question here is mixed. There are factors that unquestionably work to enhance effectiveness in an organization. But these factors are weak and defeasible, and the countervailing factors (internal conflict, divided interests of actors, slackness of corporate marketplace) leave open the possibility that institutions change but they do not evolve in a consistent direction. And the glaring dysfunctions that have afflicted many organizations, both corporate and governmental, make this conclusion even more persuasive. Perhaps what demands explanation is the rare case where an organization achieves a high level of effectiveness and consistency in its actions, rather than the many cases that come to mind of dysfunctional organizational activity.

(The examples of organizational dysfunction that come to mind are many -- the failures of nuclear regulation of the civilian nuclear industry (Perrow, The Next Catastrophe: Reducing Our Vulnerabilities to Natural, Industrial, and Terrorist Disasters); the failure of US anti-submarine warfare in World War II (Cohen, Military Misfortunes: The Anatomy of Failure in War); and the failure of chemical companies to ensure safe operations of their plants (Shrivastava, Bhopal: Anatomy of Crisis). Here is an earlier post that addresses some of these examples; link. And here are several earlier posts on the topic of institutional change and organizational behavior; linklink.) 

4 comments:

Unknown said...

As there is an upper limit on how many characters the comment can have, I will post in 2 parts. I hope that this is fine, professor.

Part 1:

Prof. Daniel Little, I may have understood it poorly, but is it implied that "evolving" means an adaptation of higher functioning? If it is so, we have a huge conceptual gap between this discussion and evolutionary theory. Especially so considering that any adaptation can be detrimental or positive concerning only some specific environments, so it is always in relative terms, and it is not the case that every detrimental adaptation will lead to immediate extinction or to any extinction at all. After all, some characteristics are detrimental or costly, and yet are selected through sexual selection.

It seems to me that the mechanisms of adaptation in organizations [agent-based, group-based, and unit-selection, as described] are comparable with mechanisms of selection in evolutionary biology: sexual selection and multi-level natural selection (gene level and group level).

* The agent-based mechanism: "One is the fact that purposive agents are involved in maintaining and changing institutional practices. Those agents are capable of perceiving inefficiencies and potential gains from innovation, and are sometimes in a position to introduce appropriate innovations".

I could argue that this is similar to sexual selection, but even more complex given the scope and complexity of human organizations. In sexual selection of social organisms one have a simpler network of prestige among members in the social group, each with their own position and possibilities of copulating and transfering their genetic information.

In an organization the number of positions tends to be higher, with clusters of similar positions designed in such way that (not considering a short-circuit caused by an informal network) information and prestige follow a determined path that enables or not network-wide changes caused by an agent.
Following the sexual selection, it is not always the case that agents consciously know that their behavior may or may not cause network-wide changes, and some adaptations may be the result of actions without any intention of doing so.

Unknown said...

Part 2:

* The group-based mechanism: "This heterogeneity of purposive actions by actors within an institution means that the direction of change is indeterminate; it is possible that the coalitions that form will bring about positive change, but the reverse is possible as well. And in fact, many authors and participants have pointed out that it is often enough not the case that the agents' interests are aligned with the priorities and needs of the organization".

I could argue that this is not only similar to group selection. It is in fact a group selection. While it is true that in some organizations the interest of the individual member isn't always aligned with what is interesting to the organization, it is not uncommon the other way around. This is why nowadays we can see this trend of firms trying to "engage" their employees, create some kind of "company culture" or increase the level of social capital among its network.

* The unit-based mechanism: "A very different possible mechanism is unit selection, where more successful innovations or firms survive and less successful innovations and firms fail. This is the premise of the evolutionary theory of the firm (Nelson and Winter, An Evolutionary Theory of Economic Change). In a competitive market, firms with low internal efficiency will have a difficult time competing on price with more efficient firms; so these low-efficiency firms will go out of business occasionally".

I could argue that this is similar to natural selection. And while I do agree that market doesn't work as selective pressure for every kind of organization, it is not the case that it needs to be the only selective pressure. Even in natural selection, different resources (food, water, shelter...) can oscillate in their availability and then change the dynamics of selection. The selective pressure also doesn't need to be the same for every organism, and indeed it is not so. Different organizations compete for different resources, sometimes with different levels of necessity between these organizations. Two organizations don't necessarily compete (McDonald's doesn't compete with the US Army, for example), and even when they do so, it may be for different aspects of what resources the object of competition can generate.

If it is so and these mechanisms of organizational change are akin to biological selection, then it seems that higher functioning must be better evaluated, for our concept of what is better may not be what makes an organization success or fail. And the capacity of an organization to succeed may or may not be what we consider that is better.

Dan Little said...

Thank you -- very thoughtful. Would you agree that those mechanisms are substantially weaker than their counterparts in biological evolution, and that there is therefore a greater likelihood of drift in organizations than species?

ZootX said...

Summarizing the strategic field, you write "his framework suggests that the structure and functioning of an organization is not wholly determined by a single intelligent actor ("the founder"), but is rather the temporally extended result of interactions among actors in the pursuit of diverse aims. This heterogeneity of purposive actions by actors within an institution means that the direction of change is indeterminate; it is possible that the coalitions that form will bring about positive change, but the reverse is possible as well."

It's interesting to see how diverse actors - suppliers, state apparatuses and regulations, customers, unions, managers, workers etc. - shape the process of organizational change. The formulation "positive change" is itself the site of contestation. What constitutes positive change?
For organizations in market economies maybe we should look at the work of Boltanski & Chiapello (New Spirit of Capitalism) and Boltanski & Thevenot (On Justification) who claim that capitalism has some essential features (capital accumulation etc.) but needs a cultural framework to motivate people. This cultural framework is not just ideology, but has a real impact on the mode of production. You can see that in the "strategic visions", "missions" etc. that organizations employ in change management and change processes to foster commitment. Successful adapting then depends on effective visions, strategies and communication processes that bind people to some course of action and reduce heterogeneity. Of course the vision and strategy itself must be also be adaptive in the given environment.