Assessing organizational capacity to adapt

Introduction

Scientific domains now largely recognize that natural phenomena at their extremes of smallness and largeness cannot, finally, be reduced to basic elements. The quantum mechanics of Max Plank was a key step in this process, leading to the development of quantum physics. Quantum physics is based on the discovery that heat and light (energy) reveal themselves in two quite different ways, as either particles or waves, but never as both at the same time. Moreover, they appear as waves if the observer is looking for waves, and as particles if the observer is looking for particles. In other words, the intent of the observer appears to be creating a relationship with the observed. Many scientists responded to this knowledge by intensifying their efforts to break particles and waves, their molecules, atoms, and their structures, into smaller and smaller pieces, in order to discover their basic building blocks, and a level at which they functioned individually and linearly. This led to the recognition among some scientists today that it is possible that the smallest particles cannot be understood as parts and that their key characteristics are their relationships and energy rather than their matter. Likewise, at the cosmic level, Albert Einstein’s theory of special relativity and then his general theory of relativity posited that the relationships between time and space are the key cosmic dynamics, rather than the mass of bodies acting independently according to defined forces and laws. These findings and their broader implications are having a growing impact on all the sciences (Kauffman, 1995; Holland, 1995; Capra, 1996; Bar-Yam, 2004; Surowiecki, 2004).

After the Second World War, a small group of scientists at the Los Alamos National Laboratory, which had played a key role in the development of the atomic bomb, began to study the phenomena of complex behavior. As their work took on a life of its own, they created the Santa Fe Institute of advanced interdisciplinary study of complexity. Most of those associated with the Institute were part time, working at universities throughout the USA and elsewhere the rest of the time. A number of renowned scientists became involved, primarily from mathematics, the natural and computer sciences, and mathematics; the Institute was multidisciplinary from the beginning. Two social scientists were also associated with the Institute, namely an economist and a cybernetics expert.

While theory about complex adaptive systems was developed largely in the domains of the hard sciences, mathematics and computers, it also incorporated the work begun during the 1940s by biologist Ludwig von Bertalanffy on general system theory (von Bertalanffy, 1968), whose work had more general applicability. He identified the autopoietic (self-regulating) character of biological systems. Work on complex phenomena grew, but remained relatively unknown and isolated until James Gleick’s Chaos: Making a New Science (1987) and Mitchell Waldrop’s Complexity (1992) popularized it. Since then, use of complex theories has disseminated quickly, and has begun to emerge as a major force in the study of both physical and social phenomena, including the fields of organizational development, administration, and management. It continues to be utilized and popularized by authors such as James Surowiecki in The Wisdom of Crowds (2004).

Because complex systems cannot be reduced to basic elements that can then be recombined in lawful ways to explain the characteristics, complex phenomena and their behaviours are studied as whole entities, systems, or patterns of behavior. They exhibit the characteristics of a great many independent agents interacting with each other in a great many ways (Waldrop, 1992: 11). Such systems exhibit spontaneous self-organization (a compulsion for order or forming patterns), which results overall in their being adaptive to their environment. While evolutionary systems are restricted to biological beings, and evolution occurs through genetic change from one generation to another, reinforced by selection, complex systems can be considered adaptive if they can change themselves using nonevolutionary processes (Holland, 1995; Kauffman, 1995; Dooley, 2004: 357).

Influenced by complexity theory, use of new methods and metaphors has emerged in the study of organizations. They include organic metaphors (Cameron & Whetten, 1983; Tushman & Romanelli, 1985; Baum & Rao, 2004), communication metaphors, and discourse methods (Abell, 1987; Czarniawska, 1998); process metaphors and methods such as life cycle, teleology, dialectics, and evolution (Poole et al., 2000: 56); and a focus on time (Abbott, 1990). Explicit use of complexity theory emerged during the early 1990s in the social sciences, although use of systems theory began earlier. A substantial literature now asserts that change and innovation emerge in complex human society and in organizations, sometimes when they are needed (Goldstein, 1994; Dooley, 1997; Lewin, 1999; Van Tonder, 2004a, b, c)

Dooley argued that a key factor in the study of organizations is time, and also, as had Glor (2006), that elements of both variance theory and process theory should be used in studying complex organizations. Systems theory has had an important influence on complexity theory, especially the notions of feedback and reinforcing loops. Complexity analyses have identified conditions under which cascading processes and exponential growth occur (Glor, 2001b; Dooley, 2004: 354, 356).

In nature, according to Rogers et al. (2005), cascading mutations/extinctions, or changes in individual species, result from variety among organisms and reactivity to change. Such changes continue in step-like punctuated equilibria that approach the critical point of self-organization. At this point a gap grows in systemic fitness. As fitness thresholds/plateaus step higher and higher, the cascades of change, with their draw on disposable resources, become larger and larger. Only those species (population categories) with sufficient disposable resources (adaptability to change) can survive at the higher fitness thresholds that occur during cascades of change. In this view, only those capable of self-organizing emerge as “the select.” The cascading continues and then it stabilizes as a complex system. In complex systems self-organized change emerges periodically and often as needed. Human systems and especially diffusion of innovation function in much the same way (Rogers et al., 2005: 5).

In studying diffusion of innovation or change, a critical threshold or tipping point (Gladwell, 2002)—a transitional inflection point associated with higher system reactivity—is observed, where system members are sensitive to change. At this juncture the system exhibits the most change for the least increase in energy. Once the rate of adoption in a system reaches critical mass at the inflection point, it is difficult or impossible to stop further phase transition around diffusion (Rogers et al., 2005: 12).

The inflection point is not always welcome in organizations. This phenomenon has been observed, for example, in epidemics, stock market crashes, and mobs. Many senior executives prefer not to have organizational change that has a life of its own; they prefer to decide it should happen. Their top-down approach allows them to maintain control of the organization—often their primary consideration—and supports a linear, limited, and controlled way of functioning. The management literature and management courses typically reinforce this top-down approach, teaching that it is the potential and responsibility of managers and executives to initiate, introduce, implement, and manage change in organizations. The preferred strategy adopted by executives for creating change is structural change through reorganizations. Reorganizations have the advantages of using tools within the scope of executives and also of destabilizing the power bases of other agents in the organization that might stand in the way of the change. The disadvantages of reorganizations as change tools are that they absorb enormous amounts of people’s energy and that they usually lead to very little functional change, despite the disruption. The newly restructured components typically settle back into behavior patterns that are similar to those that existed previously and that were sometimes integral to the problems the reorganization was meant to solve. As a result, most large organizational change projects fail to achieve their explicit goals (Grint, 1998; Applebaum & Wohl, 2000; Beer & Nohria, 2000; Mourier & Smith, 2001; Seijts, 2006), probably because they do not reach the inflection point. W. Warner Burke (2002) agreed. Moreover, following a thorough search, he concluded that there is almost no theory to explain or guide organization change (Burke, 2002: 1, 121).

Burke (2002) argued that the way to change organizations fundamentally is to intervene at the strategic level of the organization, in what he called transformational factors. The transformational factors affect all parts of the organization, and include three elements: the organizational mission and strategy, leadership, and organizational culture. On the other hand, to create continuous improvement, evolutionary or selective rather than sweeping change, Burke recommended intervening in the transactional factors. These include structure, management practices, systems (policies and procedures), work unit climate, task requirements, and individual skills, motivation, needs, and values. Structure, according to Burke, is a transactional not a transformational factor.

Glor’s approach, somewhat different from both that of the executive concerned with power and that of Burke, focuses instead on organizational contexts or patterns (in the sense used in complexity theory). By drawing on the concepts of emergent patterns and adaptation in the CAS literature, it suggests that an executive or front-line employee who wants to create real change and adaptability within his or her organization must address the patterns of functioning of the organization. Unlike much of the management literature, this approach recognizes that these are not easy to change. Nonetheless, it seems reasonable to assume that it is easier to introduce change within some organizational patterns than within others.

Organizational patterns are manifestations of the dynamics taking place within an organization, including the transformational and transactional factors discussed by Burke. They form the context for the organization. At the same time, self-organized patterns are as much the problem and can be even harder to change than Burke’s transformational factors such as mission or his transactional factors such as structure.¹

Some authors consider that the context within which change occurs is an essential element to its conceptualization (Bolton & Heap, 2002; Van Tonder, 2004b). If this is the case, then it is important to understand organizational patterns, and also their potential for change or adaptability. The model of organizational patterns (or contexts) developed by Glor (2001: a, b) for innovation are used here more broadly to apply to change generally, and criteria for adaptation drawn from the theory of complex adaptive systems are used to explore the capacity of organizations to be adaptive.

A focus on organizational patterns has a number of advantages. First, it offers a way to describe a well-known phenomenon: Despite changes to either or both transformational and transactional factors, organizations usually settle back into essentially similar ways of functioning. Second, a focus on patterns allows for an examination of the dynamics that must be present if an organization is to be capable of self-organized, emergent, complex change, the only change that is relatively certain to “take.” Third, it makes clear one of the effects that highly controlled, authoritarian organizational environments can have: Sometimes (not always) they reduce or eliminate the capacity of the organization to adapt. While adaptation may occur at the expense of power, adaptation and organizational survival are in the interests not only of employees and investors/electors, but also of senior executives, and have the potential to create enduring change.

A focus on patterns also has some disadvantages. First, the science of complex systems cannot offer easy and straightforward advice, nor guaranteed results, nor assurance for how to change patterns or produce the results wanted. While CAS are unpredictable at the edge of chaos, where the greatest amount of change occurs (Kauffman, 1995: 86-92), the dynamics also follow patterns and they can pass through more and less predictable phases. Just as forming the patterns is a complex activity, so is changing them. At the same time, linear approaches may be even less functional than complex ones and less likely to work in an increasingly complex world; already, overall, 65-75 percent of organizational change efforts fail (Van Tonder, 2004c: 33). The complex theorists argue that complex systems tend toward greater complexity (e.g., Holland, 1995, who subtitled his book How Adaptation Builds Complexity), hence increasing complexity is of the nature of organizational systems, and to be expected. Second, with organizations as with other complex phenomena, time is not reversible. Linear theories assume that it is. As a result, intervening in the factors that formed the patterns gives no assurance that the patterns will go back to their initial state or, if they do change, when this will happen. Despite the lack of assurance about the end point of change, organizations need to be more adaptable, linear approaches are insufficient, and better approaches to organizational change and to conceptualizing organizational change theory are needed. Since there is little organizational change theory (Burke, 2002: 121), this area is open to development.

Both of the fundamental concepts employed in the analysis that follows have emerged from the study of complex systems. The idea of organizational adaption is based on complex adaptive systems (CAS) theory, and the idea of the emergence of patterns of behavior in organizations was drawn by Glor from the field of study of complex systems.

In CAS theory, variety, reactivity, and the capacity for self-organized emergence are prerequisites of organizational adaptability (Waldrop, 1992: 314; Rogers et al., 2005). According to Kauffman (1995), once the prerequisites are in place, adaptation occurs “for free”; that is, without further introduction of energy or other prerequisites. It emerges. Because variety, reactivity, and the capacity for self-organized emergence are the only required conditions, they have been used as indicators of the capacity to adapt in the analysis that follows.

Methodology

Using organizational and complex systems concepts, the hypothesis is explored that organizations’ established patterns of functioning affect how and whether they can adapt effectively to their environments and are able to change when this is necessary or desirable. The potential or capacity of organizations to change within each of Glor’s eight organizational innovation patterns is examined.

Glor (2001a) expressed the wide range of relationships in the workplace through three multidimensional factors at work in government organizations. In general terms, three dynamics reflect individuals, groups, and tasks in the workplace. Arguing that their interactions lead to innovation occurring in patterns, she suggested that the dynamic interaction of individual motivation, organizational culture, and the challenge presented by implementation of an innovation produced eight organizational innovation

Glor’s patterns describe a range of organizational dynamics formed into enduring patterns as a result of the individuals and groups interacting with each other and implementation challenges in organizations. The global factors forming patterns are indicated in Table 1.

Accepting Glor’s (2001a, b) hypothesis that organizations form themselves into patterns of functioning, these patterns were treated as the context within which change occurs within organizations. The analysis attempted to distinguish different capacities for adaptation among the patterns. Using CAS as the conceptual structure for the change process, Glor’s (2001a, b) eight organizational innovation patterns were treated as organizational change patterns and examined for their adaptive capacity.

While the inputs to the factors may not be exactly as outlined in this paper in every case of change, and they may not interact in exactly this way every time, nonetheless, Glor (2001a, b) has offered some support for the idea that these factors are important, and that patterns emerge from them. Other authors have described the balance of the importance of these factors slightly differently, however. Burke (2002), for example, listed individual factors as being more important for the parts of an organization (i.e., as being transactional) and organizational culture as being important for or affecting the whole organization (i.e., as being transformational), while Glor treated the factors as having equal impact. Both showed implementation as a factor. Seijts (2006), on the other hand, identified implementation as the key issue in change. Litwin and Stringer (1968) identified similar factors to those of Glor—motivation, organization climate, and the task—as the key factors, but saw organizational climate affecting motivation that affects the task; that is, he saw them influencing each other unidirectionally. Glor treated them as interactive; that is, all three affect each other to form the patterns.

Many dynamics are at work at the individual, group, and task levels in organizations, influenced by the external environment. They interrelate and settle into dynamic patterns that have consequences for the organization. The fact that there are patterns and that there is a tendancy toward order (Kauffman, 1995) is assumed. Rather than emphasizing the inputs to the patterns, which have been explored elsewhere (Glor, 2001a, b), the consequences of the patterns for the capacity of organizations to adapt and to change was explored. The capacity of each of the self-organized organizational patterns to support adaptation and change was analyzed.

To be able to examine patterns for their likely support of or resistance to change, it is necessary to identify criteria for organizational systems that are adaptive and those that are not. According to Waldrop (1992: 314), complex natural systems that change quickly require variety and reactivity and are more likely to develop a capacity for self-organized emergence. Following Waldrop’s thinking, Rogers et al. (2005) suggested that complex organizations or social environments that change quickly have three characteristics: variety, reactivity, and a capacity for self-organized emergence, a systemic capacity to respond to the environment. Once the capacity for self-organized emergence develops, the system can become adaptive. Some ways of thinking about these criteria in organizations and suggested ways of assessing them are presented next.

Results

Using the three criteria (variety, reactivity, and spontaneous emergence) for system change to frame the analysis, and using Glor’s eight innovation patterns to represent organizational patterns, the analysis was able to distinguish different adaptive capacity among the patterns. A ranking was developed for whether and to what extent they exhibited the CAS potential for adaptation.

The reader may have observed that many of the criteria used for adaptation are taken from the outcomes of the patterns identified by Glor (2001b). This is not a problem in and of itself, but is, rather, an indication that the outcomes Glor explored related to adaptation, and reinforces the idea that the concept of outcomes as used by Glor and the concept of adaptability in CAS are related. The concept of an outcome tends to be treated in administration and management as linear (though it is not, really), while the concept of adaptation is a biological and hence complex one. The Darwinian concept of adaptation tended to be linear, however, because it relied on only one cause of adaptation, namely selection. Complex notions of adaptation recognize at least two: selection and emergence (Kauffman, 1995).

The assessments noted in tables 2 through 4 are based on assumptions, judgment, and personal experience; they are not the result of empirical measurement. Some distinctions emerged from these analyses.

The organizational patterns are assessed for the first criterion of adaptation, variety, in Table 2. The analysis detected considerable differences among the organizational patterns in terms of their variety. Three of the organizational patterns were assessed with low overall measures of variety (score = 0 to 0.3). While the active, necessary, proactive, and transformational patterns had some variety (score = 0.4 to 0.7), the continuous pattern had the most variety (score = 0.8 to 1.0).

The second factor required for change to emerge in an organization is reactivity. The reactivity of the eight organizational patterns is assessed in Table 3.

Three of the organizational patterns had low reactivity (score 0 to 0.3). According

The third criterion for adaptation, capacity for self-organized emergence, is assessed in Table 4. Within each of the patterns, the capacity for self-organized emergence has two aspects: the capacity to implement change and the capacity for changes to endure; often this means to become institutionalized. In order to implement change in an organization, approval must be secured, resources must be allocated and the change must be put in place. While a top-down organization whose management has recognized the need for change can approve a change easily, and small challenges can be overcome easily, changes that require long-term effort to implement may have more trouble. Top executives rarely give sustained attention to change.

Two of the patterns demonstrated a low capacity to implement change (score = 0 in Table 4). Three of them had a medium capacity (score of 1.0 and 1.75) and three of them had a high capacity (score = 2.0). The reactive, continuous, and buy-in patterns had a high capacity for implementation. Capacity for self-organized emergence is not solely about implementation, however, it is also about the capacity for the change to take hold, become institutionalized, and create enduring change. The patterns vary in this regard as well (Table 4). Three of the patterns had a low capacity for the changes to endure (score = 0 to 0.3), while four had a medium capacity (score = 0.6 to 1.3). One pattern—the continuous pattern—had a high capacity for changes to endure (score = 2.5).

The summed capacity of the organizational patterns to support both the implementation and endurance of changes, or emergence,

I was surprised by the finding that the reactive pattern had a medium capacity for self-organized emergence. It can perhaps be explained in the following way. The reactive pattern has a high capacity for implementation. It is quintessentially a bureaucratic environment. According to Victor Thompson (1976), bureaucracies have a high capacity for implementation (this is why they were set up), but they are also highly reactive organizations. In rapidly changing environments this has proven to be an ineffective approach, but it has also proven a useful pattern in more slowly changing environments. It is a pattern that is very hard to change. Its high capacity for implementation and medium capacity for endurance bring it into the medium category.

A standardized analysis of the capacity for self-organized emergence, created by averaging the scores within the patterns, showed the continuous change pattern, with its high capacity for both implementation and institutionalization, as having the most capacity for self-organized emergence. It was followed by the reactive and transformational patterns.

The extent to which the three conditions for adaptation—variety, reactivity, and capacity for self-emergence—are met in each of the organizational patterns is summarized in Table 5. Overall, according to the analysis conducted in this paper, the ranking of the patterns for adaptation from the least adaptable to the most was the following: imposed/active, reactive, buy-in/proactive, necessary, transformational, and continuous. The imposed/active, reactive, and buy-in/proactive organizational patterns had the least capacity for adaptation. The necessary and transformative patterns had more capacity for change than the patterns least able to adapt, and were close at the second level. The continuous organizational pattern had the most, and considerably more, capacity to adapt and change than the other patterns.

The analysis was able to distinguish among the patterns for adaptability, and basically among three different levels of adaptability.

Discussion

The analysis suggested that some organizational patterns fulfill the requirements for adaptation more than others, and that overall some organizational patterns are more fit for change than others. This is the important finding (not the levels, as the inputs were highly conceptual). If empirically confirmed, this is important information for executives, managers, front-line employees of organizations, and boards of governors/government cabinets as input to decisions about how to approach and react to change and which organizational pattern to try to build. It helps to define where an organization is starting its change initiative.

The analysis suggests that a top-down, authoritarian approach to major change may not be a very effective approach except in reactive environments, and where maintaining power at the top is a true need. While many organizations function this way, they are not the most adaptable organizations.

Under past conditions where surplus labor came mostly from former farmers—that is, where employees were mechanically skilled but often poorly educated—this approach might have had more of a place than it does today, when employees are much better educated and have similar education or even better education compared to those of their managers and executives. At the same time, both then and today, farmers, educated employees, and immigrants tend to be capable of independent action; in other words, they are not dependent people. Whether invited or not, these people bring new ideas into the organization. While a certain amount of molding to the organization may be needed, an authoritarian, rules-based environment is not the most adaptive, nor is it the best one today.

Rather, an environment in which individuals’ needs are met, they are given the opportunity to make a voluntary contribution, and implementation and institutionalization are effectively handled makes for a better work experience. It also makes for more effective adaptation and change. The current governmental, voluntary sector, private sector, and international environments require greater adaptability of organizations than in the past. While the responses to this need have often been authoritarian, an organization with variety, reactivity, and a capacity for self-organized emergence is the most adaptable.

Conclusion

This paper has applied the three conditions required for change in complex adaptive systems (CAS) to organizations. According to CAS theory, the essential elements of support for change are variety, reactivity, and capacity for self-organized emergence. Criteria for identifying the conditions in organizations were identified, and Glor’s (2001b) organizational innovation patterns were assessed for their capacity for adaptation. Variety, the first criterion, was seen as encouraged by large numbers of ideas available, large variability among the ideas, high participation rates by employees, and heterogeneous networking and consultation. Reactivity, the second criterion, was assessed by individual motivation, organizational culture, group (work unit) support of change agents, management support of change agents, and communication. The capacity for self-organized emergence, the third criterion, was seen to arise from both the capacity for implementation and the endurance (institutionalization) of changes. Together, these criteria should identify the conditions for change or adaptation in an organization.

By examining these conditions in each of the eight organizational patterns, using potentially empirical measures, the continuous change pattern was assessed as supporting adaptation and change better than the other patterns. The proactive, necessary, and transformational organizational patterns, it was suggested, would also have some but less capacity to support change. The buy-in, active, reactive, and imposed patterns had the least.

Most organizational change strategies are premised on the assumption that the organizational pattern is reactive or buy-in. Although this may be understood intuitively, it is usually not understood explicitly. Hence, this is an area where organizations could benefit from implicit knowledge becoming explicit (Nonaka & Takeuchi, 1995). David Snowden has developed some techniques such as anthropological-style observation, story circles, and analytic tools such as knowledge disclosure points and ASHEN (artefacts, skills, heuristics, experience, natural talent) to facilitate the expression and understanding of implicit knowledge in a respectful and context-specific manner (Snowden, 2000a, b, c). Those who plan change need to know which pattern they are working with in their organization; and those who wish to help their organization become more adaptable need to know both which pattern they are starting with and which one they hope to create.

At the same time, the analysis indicated that change occurs most easily in organizations where individuals are intrinsically motivated, the culture is nonhierarchical, and challenge is minor. Necessary change will not be difficult to achieve in such an environment. Change will occur least easily in environments in which individuals are extrinsically motivated and working in a top-down organizational culture. They are the least likely to self-organize to create change.

Hierarchy, which characterizes most large organizations, represents a barrier to both innovation and change. Managers in many organizations in effect deliberately reduce system variety and reactivity, build barriers to heterogenous interaction, and retard the self-organization of change, in favor of control. Their objective appears to be to ensure that complex adaptative systems cannot develop. This retardation of adaptation can be done, for example, by placing a great deal of power and authority at the top of the organization, by punishing unauthorized activities and motivations, and by giving employees so much work that they have no time to interact with their environment (other employees, peers, or friends). This can be an effective formula for the creation of linearity. The organization cannot adapt well and will face regular crises. Because it does not develop the intrinsic motivation and the bottom-up skills of the members of the organization, employees do not have the skills or the flexibility to function in a manner that allows for or encourages emergence.

An organization that does not have a good capacity to change even when change is needed typically introduces change in a linear, power-based, no choices allowed, mechanistic manner. As suggested by the analysis, change can occur in such an environment, but the organization relies overly on the knowledge and judgment of its executives. Despite the reliance of organizations on their executives for making decisions, there is no evidence that they make better decisions (as revealed by quality initiatives and Surowiecki, 2004) than other employees. Moreover, self-organized, emergent change is not likely to occur.

The analysis suggested that some of the organizational patterns have little capacity for change, others some potential, and one has a great deal. The conditions defined for encouraging emergence are thus intrinsically motivated employees, a bottom-up culture, many ideas, support within the unit and by management, communication outside the organization, and reduction of the challenge associated with implementation (decentralizing the power to approve staff and for innovations and changes, for example). The conditions for discouraging emergence, adaptability, and change include extrinsically motivated employees, a top-down organizational culture, few ideas, little support, lack of communication outside the organization, and major challenges associated with implementation. While emergent change does not enhance central control, it does enhance adaptability.

Many issues should be further researched and discussed. The analyses conducted in this paper need to be mathematically modeled and empirically tested. An important question for further exploration is, if an executive is aware of the organizational conditions of support or lack of support for change in his or her organization, what can he or she do to take advantage of the potential or to develop a more adaptive environment? If an organization wants to become more adaptable, how can this be done?

Another area where work is needed is on social and organizational theory to explain how and why the individual, the social, and the challenge influence each other in organizations, and how and why the existence of the requisite phenomena for evolution cause new human phenomena to emerge. This is a general problem in studying individuals, social phenomena, and processes involving humans (Coleman, 1986; Burke, 2002). While the line of reasoning and study presented here for the study of organizational change seems promising, it does not yet have a solid theoretical base. While a causal chain is not required in evolutionary and complex theory, better understanding of the interaction among individual, social, and organizational processes is necessary.

Note

1. Structure is about differentiation, while function is about integration (White, 1959: 144).