Innovation in organizations from a complex adaptive systems perspective

Introduction

A strategic choice perspective views innovation strategy as a function of managerial choice in the light of environmental contingency. Writers like Miles and Snow (1978) and Freeman (1982) provided taxonomies of strategies to identify the types of strategic actions a company might adopt. Commercialization of promising innovative developments requires a mix of profiling, marketing, and promotional strategies, but the idea that effective innovation can depend in the first instance upon a carefully planned long-term strategy has lost force in less stable and less predictable environments.

Learning has been important in the strategy literature (e.g., de Geus, 1997, 1998; Mintzberg, 1987; Senge, 1990). Strategy has been conceived as a process of learning and innovation in the face of environmental change (e.g., Burgleman & Grove, 1996). This conception appears to reflect the reality of requirements in the face of changing environments (e.g., Tichy & Charan, 1989; Bartlett & Ghoshal, 1993; Burgleman & Grove, 1996) more accurately than the top-down formal planning and control models of earlier years. Strategy schools which could be described as largely top down or prescriptive (e.g., Porter, 1980, 1985) have been argued to undervalue learning and overemphasize control (Mintzberg, et al., 1998). In stable conditions, with higher degrees of control, learning is a narrowing and converging process of testing. In chaotic conditions it is a process of expansion, divergence and discovery (Cheng & Van de Ven, 1996).

Mintzberg (1978) noted that strategies have both emergent and deliberate features, a thesis he later developed with Waters (Mintzberg & Waters, 1989). Quinn (1978) noted that companies often progress from general ideas through to specific commitments in an incremental even ad hoc fashion. He pointed out that effective strategies are often flexible and experimental and that their coherence is often not derived from formal strategic planning and control. Mintzberg’s (1987) elaboration of the deliberate/emergent strategy distinction subsequently highlighted how the deliberate and emergent dimensions of the strategy process are linked to control and adaptive learning, respectively, and combine to produce strategies which are crafted rather than planned. However, this does not imply that the convergent learning associated with control and stability takes place in a balanced proportion with the divergent learning associated with instability and autonomy.

In turbulent environments, innovation, radical innovation in particular, has been argued to be a strategic imperative. Some commentators stress the need for radical ‘framebreaking’ discontinuous innovation at the expense of incrementalism. For example, Tom Peters (1999) believes that “incrementalism is innovation’s worst enemy” (Peters, 1999: 27). He suggested that “the only sustainable competitive advantage comes from out-innovating the competition” (Peters, 1999: 29) and that “the bottom line is: Innovate or die” (Peters, 1999: 308). Gary Hamel (2000) echoed this in the preface to his book Leading the Revolution. He claimed that “radical innovation is the competitive advantage for the new millennium” and like Peters believes that “we’ve reached the end of incrementalism” (Hamel, 2000: xii). Eisenhardt and Martin (2000) have pointed out that environmental turbulence has, in many industries, led to shortened periods of stability and incrementalism and more frequent periods of radical change, but this does not mean that incrementalism is dead. Getz and Robinson (2003) highlight how adherence to the ‘Innovate or Die’ mantra leads managers to neglect other important sources of competitive advantage.

We step back from the more polemical accounts of strategy and innovation, which are evident in the literatures, and from any idea that radical and incremental innovation may be alternatives. We contend that both types are needed for organizational success together with a coordinated approach to organization management, but we question the notion of ‘balance’.

Exploitation and exploration in the process of innovation

A case for a balanced strategic approach to innovation and change has often been made. March (1991) identified both exploitation and exploration as essential. Short-term financial performance involves exploiting existing knowledge, competencies and capabilities, making incremental improvements to sustain competitive edge. Longer term survival involves exploring new knowledge and new environments in which to build new competencies and capabilities for the future. The implication is that these activities need to be balanced to sustain current performance and achieve organizational renewal (March, 1991; Levinthal & March, 1993; Volberda & Baden-Fuller, 1999). Exploration requires the adoption of a long-term strategic perspective whereas exploitation requires a short term view of how to make the most of existing competencies through incremental improvements.

The polemical stance towards radical innovation can be viewed in the light of evidence that many firms prioritize exploitation at the expense of exploration (e.g., Leonard-Barton, 1992). They fall into ‘the competence trap’ (Leonard-Barton, 1992) in which existing competencies become obsolescent before new ones have been developed. Many firms are more effective as incremental innovators, but in the longer term, they need to engage in both divergent and convergent learning, radical and incremental innovation, exploration and exploitation (Cohen & Levinthal, 1990; Kogut & Zander, 1992; Eisenhardt & Martin, 2000). But do these approaches need to be balanced?

Complexity theory suggests to us that the idea of ‘balance’ is problematic. March (1991) focused on trade-offs between exploration and exploitation, and the notion of reaching some kind of ‘optimum’ ‘balance’ or mix, but a static balance is not suited to rapidly changing environments. Any ‘optimum’ would be short-lived. The concepts of balance and optimum may imply an equilibrium. It is worth noting in this context, that despite March’s (1991) use of the term ‘balance’, he alluded earlier (Levitt & March, 1988) to its problematic nature. Incremental innovation geared to efficiency may drive out radical innovation making the two incompatible. March (1998) has since re-affirmed his view that a balance, as conventionally understood may be impossible to achieve.

Complex adaptive systems and innovation

Complexity science provides a fresh perspective on organizations and organizational activities. Although developed in the natural sciences, its findings have managerial implications in the organizational sphere (McKelvey, 1997; Pascale, et al., 2000; McMillan, 2004; Stacey, 1992, 1993, 1996; Levy, 1994). From the complexity standpoint, organizations are dynamical systems. They are complex adaptive systems comprised of agents (people) who experiment, explore, self organize, learn and adapt (in varying degrees) to changes in their environments. They exist at the individual, team, divisional and group level and also in a much larger web of external complex adaptive systems—their economic, social and political environments. Complex adaptive systems are everywhere in the natural world ranging from relatively simple organisms like viruses to ant colonies, to more sophisticated creatures like mammals. Complex adaptivity could be described as a successful evolutionary response to the survival needs of certain species. Their adaptive capabilities delineate them from systems which are merely complex. People as individual complex adaptive systems are adept at self-organizing; at manipulating their environments; at turning things to their own advantage; but most of all at learning and adaptation. Their ability to learn and adapt is underpinned by key self-organizing behaviors including exploration and experimentation.

Complex adaptive systems constantly seek to adapt to the environmental circumstances in which they find themselves. Thus they are able to undertake short term exploitation activities as required and to invest in longer term exploration as needed. Their activities are determined by the conditions in which the system finds itself, and it responds in a self-organizing reaction. A CAS does not ‘differentiate’ between the long term and the short term - it simply self-organizes appropriately. It engages in multiple activities of different types. The message here for organizations is not to take too rigid a stance in approaches to innovation, but to respond flexibly as internal and external environments demand.

We can place organizations along a spectrum, ranging from random, unorganized and highly chaotic to highly ordered and mechanistic. Figure 1 shows the relationships between different types of organizational systems of order and control, in relation to the properties of each system and their position on the spectrum.

At one end of the spectrum, random and chaotic systems are highly unstable. At the other, mechanistic and hierarchical systems are highly stable and ordered. In the middle lies the complex adaptive system behaviors. Brown and Eisenhardt (1997) illustrated how the region of emergent complexity lies between the regions of order and chaos. It is often argued that at ‘the edge of chaos’, represented on the diagram by the line dividing the complex and chaos regions, conditions are right for both single and double loop learning (Morgan, 1986) and for radical innovation. At ‘the edge of chaos’, firms are most likely to develop successful novel survival strategies because expansive and exploratory learning leads to new discoveries. These present fresh choices that may in turn lead to new options for innovation and survival. In highly settled orderly conditions learning is reduced to a limited process of testing. These conditions are right for exploitation and can lead to efficiencies with short term value, but over time restricted experiences can narrow the range of options a system can conceive of as future possibilities for action. Complex adaptive systems may on occasions veer towards the edge of stability, represented on the diagram by the line dividing the complex and hierarchical regions, but they seek to remain in the region of emergent complexity.

The survival of a CAS is not certain, but in rapidly changing environments it will try to increase its chances by seeking to operate primarily at ‘the edge of chaos’. This is explained by Fisher’s (1930) theorem: the more genetic variance in a species, the more likely it is to evolve into new niches. In organizations, competitive advantages are sustainable in turbulent environments only when adaptive rates rise (Barnett & Hansen, 1996). Sustaining short-term performance can mean increasing the pace of incremental change, securing long term survival may mean simultaneous radical innovation. We suggest that organizations need to maintain internal variance between different departments and units, some focusing on faster incremental change and others on more radical innovation.

Adaptive capabilities in complex organizational systems can be inhibited and damaged by the conditions in which they exist. Very stable and orderly zones have been likened to the internal environment frequently found in large, traditional bureaucracies (McMillan, 2004). Established research on creativity and innovation suggests that rigid procedures, bureaucratic regulations and hierarchical controls hamper creativity and open communications (see, e.g., Amabile, 1988; Woodman, 1995). At the other extreme where internal environments are too unstable and disorderly they risk disintegration. This was the fate of some of the 1990s dotcoms. Management processes and cultures are needed to enable organizations to stay within the emergent complexity zone. This means emulating a CAS by constantly learning from experiences, recognizing these, continuously adapting to this learning and to internal and external environmental change. This calls for flexible and responsive frameworks and management processes, and a culture which encourages learning at both single loop (testing) and exploratory (potentially double loop) levels as required. Evidence suggests that incremental innovation is more likely to occur when control mechanisms veer towards the tighter end of the organizational—bureaucratic spectrum, while radical innovation is more likely in looser less tightly controlled conditions (Eckval, 1996).

Stacey (1995) noted, in considering a competing population of complex organizations, that some, given a primary task to innovate would operate close to ‘the edge of stability.’ Some of these could be expected to be successful in developing important strategic outcomes by incremental means. Others would be unsuccessful and weeded out by competitive selection. Of those operating close to chaos and instability some could be pulled into failure by unstable disintegrating forces. Others would succeed in radical innovation by ‘dancing’ on the edge of these extremes. There is no guarantee of success, but the real danger lies in being pulled too far by the extremes of stability (ossification) and instability (disintegration).

We suggest that different parts of an organization need to operate in different parts of the spectrum to respond and adapt to internal and external change, adjusting their responses in line with valuable learning experiences. For example, an accounts department may operate most of the time close to stability, but not so close so that it fails to respond to needs for new practices and procedures reflecting changes in production, sales or research. A sales and marketing department in the same organization may operate close to instability as it launches an experimental and radical advertising campaign. When an organization operates at ‘the edge of chaos’, the different forces within it pull it towards this area. Too many traditional, slow moving and unresponsive departments or units engaging in only single loop learning or testing, may drag the whole enterprise into the stability zone. The converse is also true. Too many areas experimenting frantically and creating huge waves of novelty can overstretch the organization and draw it into instability. This suggests that senior managers need to know where different parts of their organization are operating with regards to the emergent complexity zone and to ensure that it is not pulled to any extreme. To achieve this they will need to observe, know and understand the workings of different parts of the organization and to appreciate it as a diverse entity and a complex, dynamical whole. They will need to provide strong frameworks to guide their organizational practices and ensure robust processes for accountability, responsibility and decision-making—along with a culture that encourages and acknowledges the value of learning, experimentation and adaptability.

Innovation and environment

Managers cannot control environments, but they can influence the shape of what is referred to in complexity theory (adopting the term from biology) as ‘the fitness landscape’. The fitness landscape is in effect the environment in which organizations compete. Their actions and emergent strategies impact upon the coevolution of this environment—an implication which resonates strongly with the literature on organizational change. This recognizes that organizations do not merely respond passively to external changes but contribute to environmental change by their actions.

Complex adaptive systems like all living systems are connected or seek to be connected. Lewin (1993) refers to the importance of connectedness in evolution and points out that in living systems, ossification occurs without it. Organizations, if they are to operate as CAS, need to ensure that they connect both externally and internally. Externally, organizations need to foster excellent communications and links to the diverse environments in which they exist and which they influence. Internally, effective communications foster accurate unimpeded information and intelligence flows around the enterprise. Kauffman’s (1996) studies of search on ‘rugged landscapes’ is suggestive of when it is likely to be beneficial for an organization to focus more of its innovative resources on the search for radical innovations and vice versa. He suggests that when fitness of variants on a landscape is average, it is best to search for new possibilities a distance away from the space of possibilities in the current space. In innovation management terms, this suggests that, in an environment in which it is becoming increasingly difficult to sustain competitive advantages for any time by incremental improvement, prioritizing the search for new possibilities, which are likely to involve radical innovation, could be desirable as this would move the organization into a new competitive space of possibilities. Once the firm has moved into this space, it may be desirable to prioritize incremental improvement in order to stay ahead of the game. Kauffman suggests that as fitness increases, this new space of possibilities will attract more of the fittest variants on the landscape. In organizational terms this suggests that, as an organization fine-tunes and enhances its competencies capabilities and market offerings, it will face increasing competition in that space until competitive advantages erode. This is clearly an ongoing process of coevolutionary change which, interpreted in organizational terms, will involve the organization in ongoing radical and evolutionary change and innovation with different priorities at different times.

Fostering innovation: Dancing on the edges

Kauffman (1996) in his search for conditions offering ‘order for free’ on the complexity zone offered a metaphor for organizations as patchwork quilts, consisting of connected patches. The patches must be able to act selfishly in relation to one another, but they must also interact and communicate. Problem solutions which are effective in one patch will impact upon the problems faced by others and the parts of the patchwork system will coevolve in relation to each other, and the environment in which they operate. In relation to this framework, organizations in which self-organization is fostered might be expected to consist of ‘patches’ operating between stability and chaos.

The concept of ‘patches’ can be criticized (e.g., McKelvey, 1999) as it does not provide an image of fluid groupings and lacks the dynamism of some more recent studies. Miles, et al. (1999) consider organizations as ‘cellular networks’. Cells of self-organized employees networked with other cells, individuals and organizations which are not hampered by excessive controls or bureaucracy, but are still entrepreneurially responsible to the larger organization. These cells are fluid in that they are expected to evolve disband and re-form as circumstances require. In their description of these cells, they highlight one of the problems with many human organizations that there is a tendency over time for rules, procedures and processes to become overly complicated (Fuller, 1985). Whether we consider cells or patches, drawing upon Stacey’s (1995) analysis discussed above, we can identify four possible outcomes for coevolving groups concerned with innovation (see Table 1).

Large established organizations may need to move away from traditional, cumbersome bureaucratic processes if radical innovation is to emerge. Simplifying procedures and processes and increasing employee involvement in decision making and strategic activities is a way forward as McMillan (2004) demonstrated. In ‘the edge of chaos’ zone controls and rules are kept to a minimum and agents are given as much scope and support as possible to self-organize into ‘cells’ or groups and to network. At the same time, the complex zone, as Figure 1 shows, is not a random zone without structure. Participants need a degree of encouragement and guidance to self-organize in appropriate ways. This is one potential purpose for practices as career development and staff appraisal.

However, in some areas at some times, when success in incremental innovation is an important if not more important than radical innovation, too much freedom can be a limiting factor. As Szulanski and Amin (2001: 546-547) point out, too much freedom in some areas at some times can be problematic in human organizations. It can lead to a sense of chaos if not actual chaos, generate too many ideas and even lead to an organization losing touch with reality.

Elaborating upon Stacey’s work, we suggest that success requires cells to operate sometimes and in some areas towards ‘the edge of chaos’ and at other times in other areas, closer to ‘the edge of stability’. They need to ‘dance’ on the edges without falling out of the emergent complexity zone. Organizations may then veer between the two extremes of this zone as circumstances require.

McKelvey (2004: 5-6) citing a 12 year study of a cosmetics firm by Thomas, et al. (2005) notes the tendency in UK and US literatures to treat the exploitation—control/exploration—autonomy issues surrounding innovation issues as a duality. He notes that in France, they are not considered as opposing forces to be ‘balanced’, but as bi-polar entangled or intertwined forces. He notes that Thomas, et al. (2005) found that the rate of ‘oscillation’ between the control (edge of stability) and autonomy (edge of chaos) poles within the complexity zone is unstable. Thomas, et al. (2005) make the following suggestions for management practice:

Conclusions

In this paper, we have adopted a complex adaptive systems perspective in suggesting that organizations will need to ‘dance’ between ‘the edge of chaos’ and ‘the edge of stability’ if they are to create a sustainable innovation advantage. To achieve this managers need to set this dancing dynamic in motion and to encourage a mix (but not necessarily a balanced mix) of radical and incremental innovation focussed activities. Published complexity-informed work offers some suggestions about how managers might be able to foster conditions amenable to simultaneous exploitation and exploration activities in this way. Complexity in organizational studies is still a comparatively new field of study in which there is a need for more published empirical research, especially in relation to large organizations. How to operate as a successful complex adaptive system and to dance freely without becoming dangerously entangled in one extreme state or another is not an easy question to answer. Nonetheless, we hope we have been able to indicate that some complexity inspired pointers are emerging.