Wallis, S. E. (2020). Evaluating and improving theory using conceptual loops: A science of conceptual systems (SOCS) approach. Cybernetics and Human Knowing, in press.
This paper is based on a presentation titled “Existing and Emerging Methods for Integrating Theories Within and Between Disciplines” at the 56th annual meeting of the International Society for Systems Sciences (ISSS). July 15-22, 2012, at San Jose State University, California
I appreciate the excellent insights of three anonymous reviewers whose suggestions have led to an improved paper. All remaining mistakes I claim as my own.
When creating theory to understand or implement change at the social and/or organizational level, it is generally accepted that part of the theory building process includes a process of abstraction. While the process of abstraction is well understood, it is not so well understood how abstractions “fit” together to enable the creation of better theory. Starting with a few simple ideas, this paper explores one way we work with abstractions. This exploration challenges the traditionally held importance of abstracting concepts from experience. That traditional focus has been one-sided—pushing science toward the discovery of data without the balancing process that occurs with the integration of the data. Without such balance, the sciences have been pushed toward fragmentation. Instead, in the present paper, new emphasis is placed on the relationship between abstract concepts. Specifically, this paper suggests that a better theory is one that is constructed of concepts that exist on a similar level of abstraction. Suggestions are made for quantifying this claim and using the insights to enable scholars and practitioners to create more effective theory.
I begin with the view that the social sciences have not yet developed theories that are inarguably highly effective in practical application. One reason for this lack of progress is that we have not yet developed an adequate understanding of conceptual systems such as theories, models, and policies. Lacking that understanding, the social sciences has instead heeded calls for standards such as parsimony, creativity, compelling authorship, and bravery. These pursuits, while useful to some extent, are clearly insufficient to our task of creating theories that are inarguably highly useful in practical application.
This is not to say that creativity is not useful. Indeed, it is necessary to the process of science (c.f. Ambrose, 1996; Stacey, 1996; Thagard & Stewart, 2011; Uzzi & Spiro, 2005). While the creation of innovative concepts is needed, it is not always clear how those innovative concepts may best be interconnected to form theory that might be more useful in practical application and/or research.
The science has also expended considerable effort in the pursuit of empirical evidence; again, without the creation of effective theories. Indeed, there may be a negative effect—because the creation of so much data may have impelled researchers toward ever more narrow sub-specializations (Phillips & Johnstone, 2007); thus accelerating the fragmentation of the sciences. Therefore, it seems that we need new understandings of how to create and evaluate theory so that we may fulfil the dream of the social sciences and develop theory for supporting and transforming our social systems.
While the present paper is focused on the social sciences, these same insights may be generalized to all sciences. This is because all forms of science and investigation use concepts, and all those concepts have some level of abstraction.
Here, I use the term “theory” in a broad sense to refer to a conceptual system that is similar to a model, schema, policy, or other set of interrelated concepts that may be used to understand and/or engage the world. Such concepts and conceptual systems may be understood as existing in a conceptual world that is related to, but distinct from the real world of physical objects. Here, I am addressing concepts and conceptual systems themselves—not delving into existing “conceptual systems theory” that might be applied for understanding learning and other behaviors (Fedigan, 1973). That is to say, I am primarily looking at the concept-to-concept relationship rather than the concept-to-reality relationship.
A concept may be understood as a named abstraction from a pattern recognition experience; and, a set of concepts may be understood as existing in some systemic relationship—much as reality is systemic (James, 1909). The way that concepts fit together allow our minds to grasp related ideas, understand things, and may even be necessary for thought, itself (Fodor, 1998). Because our reality is systemic, developing an improved understanding of how our concepts are systemically interrelated provides us with a better understanding of our world. Indeed, there is a general consensus that theories which are more systemic will be more useful (Dubin, 1978; Friedman, 2003).
Indeed, if we return to the concept of Plato’s cave, we must recognize that we will never be able to understand “reality”—that the best we can do is to understand the systemic interrelationship between the shadows. Thus, understanding concepts in systemic relationship to one another seems to provide a new way to define (or at least understand) those concepts as they may be perceived or understood (perhaps creatively) without resorting to claims of what might constitute some absolute yet unreachable reality.
In this paper we will be looking at the similarities and differences between objects within and between those worlds because this is an area that requires deeper understanding if we are to make more progress in our science (Wallis, 2008). The present paper addresses ideas that are important to studying theories as conceptual systems in ways that are similar to the way we understand physical systems, biological systems, and social systems.
Increasingly, our understanding of the structure and complexity of conceptual systems is providing us with more insight into how we can develop more effective conceptual systems for more effective practice. While this kind of understanding has been suggested in the past (e.g., Dubin, 1978; Kaplan, 1964) it was not approached with scientific rigor or quantitative analysis. Building on insights from complexity theory, systems thinking and Axelrod’s work into concept mapping, recent advances in this direction include insights into the systemic structure and complexity of theory (Wallis, 2010a), policy (Wallis, 2011) and ethics (Wallis, 2010c). Others engaging in similar studies have quantified the complexity and systemicity of conceptual systems for learning (Curseu et al., 2010), studies of integrative complexity to support cognitive processes and management strategy (Tetlock, 1985), narrative analysis (Greenhalgh et al., 2005), program evaluation (Rogers, 2008), and casual mapping for organizational change (Ackermann & Eden, 2004).
One kind of conceptual system, social theory, has not been shown to be highly effective. Broadly, social change theory seems to have failed (Appelbaum, 1970; Boudon, 1986). In the field of organizational change, high failure rates are associated with the application of Total Quality Management (MacIntosh & MacLean, 1999), culture change (Smith, 2003), and Business Process Reengineering (Dekkers, 2008). Also, there is a failure of organizational theory (Burrell, 1997) and theories of bureaucracy (Bernier & Hafsi, 2007). Despite its current popularity, management theory might be causing new problems as it tries to solve others—such as the collapse of Enron (Ghoshal, 2005). Even the vaunted discipline of economics has not proven very successful (Dubin, 1978, citing Rapoport). We are not sure if it is even possible to make theory (Kessler, 2001) and some have incited us to ignore theory (e.g., Shotter, 2005).
In the present paper, I focus on the idea of abstraction. What is an abstraction, where does it originate, how might it be manipulated, and how might abstractions be best used to create more effective theories. By understanding relationships between abstract concepts in a new way, we may learn to understand conceptual systems in a new and more useful way. Indeed, this approach may be understood as supporting broader efforts at understanding and integrating conceptual systems to create more effective theory for more effective application (Wallis, 2008, 2010b).
Starting with the basics, Quine (1980) draws on Russell’s theory of types to discuss abstraction in terms of “individuals” on one level, “classes of individuals” on another level and “classes of classes” on a still higher level of abstraction from concrete to abstract. Leading him to ask, “How much of our science is merely contributed by language and how much is a genuine reflection of reality?” (Quine, 1980: 78). Drawing on Korzybski’s work (which was developed in more depth by Hayakawa, who describes the process as moving between levels on a ‘ladder of abstraction’), it is noted that “Moving up the ladder entails looking at similarities and ignoring differences” (Seabury, 1991). Indeed, on each level, some details may be gained or lost thus changing one’s view of (what may be considered as) ‘reality’.
It may be that the notion of reality is not as useful as some believe. Fodor (1998) argues that each concept is atomistic—without structure. Although, of course, each concept is an abstraction of a whole or part of an object in the real world. Already, we have an interesting distinction between the conceptual world and the real world. In the real world, an object such as an apple is systemically interrelated with other things (e.g., tree, bird, farmer, water).
Yet, in the “isolation chamber” of a person’s mind, it is possible to consider the concept of apple separately from those other things. Indeed, we may also consider (in our minds) that concept of an apple as existing in closer proximity with other concepts—such as the Andromeda galaxy. In contrast, distant galaxies arguably have negligible effect on real world apples. So, the systems of interrelationships we create in our minds do not necessarily reflect the systems in the real world. This is much the same as we are able to imagine a perfect circle—where none exists in the real world.
This is of great importance because systems scientists have spent considerable time and effort studying real world physical systems (e.g., the interaction of atoms within the apple), biological systems (e.g., the growth of the apple), and social systems (e.g., farming communities and recipes for apple pie). In contrast, scientists have spend very little effort studying conceptual systems. We’ve been stuck looking at relationships between things and thoughts; we’ve not been considering the relationships between thoughts and thoughts.
It is generally accepted that the process of description and abstraction is one way to build a theory (Morgeson & Hofmann, 1999; Ostroff & Bowen, 2000). It is also interesting to note that theories may be developed through the use of data that is derived from differing levels of abstraction. For example, Newton used observation to gain empirical data to build his theories of motion. Einstein, on the other hand, used data in the form of existing theories to create a more effective theory (Dubin, 1978).
A highly interesting thought here is that the empirical observations by Newton led to a useful abstract theory. In contrast, the integration of theories led to Einstein’s theory that was more useful than Newton’s. This is not to say that their work was solely on one side of the scale or the other. Only that one emphasized theory and one emphasized data more than the other. On a side note, one can, of course, see theory as data! However, this difference (even a difference of scale) suggests an important opportunity to use existing abstract theories in a process of rigorous integration to create theories that are more effective than we ever imagined possible. In short, the present conversation suggests that the social sciences should follow in the footsteps of Einstein rather than Newton.
Let us return now to the idea that nothing exists in isolation. This is analogous to the idea that “everything in the world is interrelated/interconnected/connected” that everywhere we look there are structures with “interaction patterns” (Axelrod & Cohen, 2000: 6). I suggest that this rule, which applies to objects, humans, and societies, also applies to concepts within our minds. Such an approach is foundational to the systems sciences where a system is, “A structure of interrelated elements” (Daneke, 1999: 230). Also, where our understanding takes, “into account the interrelationships and interdependence between the parts of a system” (Hammond, 2003: 11). Indeed, “a theoretical term has a systemic meaning” (Kaplan, 1964: 57—emphasis in original). Therefore, it follows that nothing can be usefully understood except in relation to other things.
As we delve into this level of nuance the questions emerges, what kinds of things should be placed in relationship to things in order to improve the usefulness of our understandings? My assertion, and the focus of the present paper, is that we generate more understanding and so things are better understood when they are seen in relation to things that have greater similarity to one another. This is the key—that greater similarity opens the opportunity for greater understanding. It does not seem to be so important that this similarity occur at any particular level.
It may be, however, that a greater similarity of levels results in greater insight for application while a greater difference in levels (as in highly imaginative or speculative connections) may result in more artistic insights. What traditional science might call “misinterpretation” we might call “alternative interpretation.” That alternative allows for creativity, imagination, and insight—critically important features of life, research, and science. Clearly, there is the opportunity for more investigation on this topic. For the time being, the present paper, however, is focused on abstraction.
In contrast, things that are less similar do not lend themselves quite so well to a process of improved understanding (although those differences may open the door for greater metaphors—but that would be another investigation). This is important because we use our theories to generate understanding. With more understanding, we are more able to enact change in our organizations and communities.
For our discussion, let us start with an example from the physical world. Here, I will use examples that are easily accessible to readers from a wide variety of disciplines.
If we compare apples and oranges, we are talking about a class of things called apples and a class of things called oranges. Let us assume, for the purpose of this presentation, that we have little or no experience with these fruit. We may look at an apple and say, “It has color.” We may improve our understanding of the apple by placing it on a table near an orange. In comparing the two fruits, the apple is seen to be more red and the orange to be more recognizably orange. By comparing the two, we can more easily identify differences in flavor, color, texture, etc. Indeed, such comparisons may well arise in the reader’s mind at this moment.
However, such a broad difference indicates that we can judge/compare/understand apples only broadly. When I write “apple and orange” I suspect that most people will imagine one apple and one orange—a representative sample, if you will, or perhaps an idealized expression. Few minds will immediately conjure all of the many varieties of both fruits! Yet, there are many variations in flavors among all those different apples.
To improve our understanding of apples, we may remove the orange from the table and place the apple in proximity to another apple. For example, we might compare a golden delicious (that is more yellow in color) with a pippin (that is more green in color). Such a distinction among apples is not possible when comparing apples with oranges. This one brief thought experiment can be repeated successfully with many objects. Within the classroom, for example, comparing chairs to walls yields superficial differences. Comparing chairs with other chairs, however, impels students to identify more details.
Moving up some scale of nuance in our comparisons, we could compare two apples of the same kind. For example, two pippin apples. To an individual with a high level of discernment, it would be possible to detect slight differences between two pippin apples (variations in color, flavor, etc.). Those subtle differences are not so evident when comparing pippin apples (or apples generally) with oranges. Zooming / graining back out to compare objects of greater difference instead of less, apples are less well understood in relation to (for example) automobiles. A car might have an “apple red” color. Which provides one point of comparison. However, there is not a good way to consider the difference in structure and function between apples and cars. There are few points of comparison. What are car seeds? No such thing.
How about comparing apples with stellar nebula? The greater the difference, the fewer opportunities exist for comparison. Moving further away, it would be more difficult still to compare an apple with a highly abstract concept such as freedom. Freedom, does not have color, or flavor in the same way that an apple does.
Here, I am not saying one cannot find any comparison. As a side-note, another possible use of identifying similarities is to create abstraction. If we place apples and galaxies in the same category, we would be impelled to identify some similarity between them. That similarity, the name of the category, becomes a new abstraction. For example, apples and galaxies are both made of matter. My point is that there are fewer relationships and so fewer opportunities to identify nuances of difference. Thus, fewer opportunities to generate insight (including creative insight) and understanding.
Those differences, if one is impelled to find a connection, may generate more metaphors. And, indeed, we might span that gulf by the use of metaphorical relations and other poetic endeavors. While those too may be interesting, they are not useful for practical application. Indeed, they are likely to lead to misleading speculative endeavors. For example, one might “reach out and pluck an apple” but reaching out and plucking a galaxy is not likely to be a successful action or result in a tasty snack.
This is not to say that metaphors are without merit. Indeed, they can be a great spur to imagination and insight. Those benefits have been explored by others (e.g., Baake, 2003; Daneke, 1997; Fuller & Moran, 2000; Hatch & Yanow, 2008; Hung, 2002; Kuipers, 1982; Lakoff & Johnson, 1980; Letiche & van Uden, 1998; Morgan, 1980; Robbins, 2000; Speicher, 1997) and experienced by all of us. Here, however, I seek to identify useful approaches from a different direction. In short, while metaphor is certainly useful, “we are after bigger game than metaphor” (Wallis, 2009b: 86).
The usefulness generated by similarity extends in other directions as well. If one learns the appropriate technique for picking one type of apple from a tree, one may easily learn to pick other types of apples. Different techniques are required for picking oranges, grapes, peanuts, and so on. The further one drifts from the original, the less useful the insights are in practical application.
When comparing concepts, the same approaches and insights apply. That is, we are able to gain a more nuanced understanding of a concept when we compare concepts that are more similar. We gain more knowledge when we compare concepts that are closer together. Here, I would like to focus on the idea that those differences are particularly evident when the conceptual objects exist at different levels of abstraction.
For example, it is not very illuminating to compare the more concrete concept of apple with the more abstract concept of fruit. This may be (at least in part) because apples are a sub-set of fruit. Therefore, all concepts that are part of the apple are already included in the more abstract concept of fruit. Yellow apples, green apples—all are fruit.
For an applied example, if a child is learning basic math functions, the idea of multiplication is easier to learn if the child has already learned addition because the two concepts are similar (four times three is the same as four plus four plus four—or four plus four, three times). Learning the concept of multiplication would be much more difficult if the student were to begin with the concept of glassblowing because the concept of glassblowing is not as well connected to multiplication as the concept of addition.
For another example, one might compare the concept of “movement” with the concept of “size.” There is not much overlap there—they seem quite distinct. However, if we compare the concept of “upward movement” with the concept of “lateral movement” clear similarities and differences begin to emerge. We are more easily able to acquire useful understandings.
In academic circles, in creating theories for the social sciences, we have attempted through the centuries to define and clarify the apparent connections between things (objects in the physical world) and concepts (objects within our mental world). This empirical (or, perhaps, positivist) approach has not met with great success. There is no evidence that the theory-based practices of the social sciences are more effective now than they were 50 years ago. Indeed, practitioners call loudly for useful theories (when they bother to look our way at all) and studies across a range of fields show that our theories are of very limited value (as noted above).
To summarize, more useful understanding may be gained by comparing things that are more similar. That is to say, if we want to make effective theories, it seems that they should be constructed of concepts that exist at the same level on some scale of abstraction. That approach optimizes the opportunity for insights to emerge that will add to our knowledge and the usefulness of the theories. Or, to phrase it a bit differently, a conceptual system such as a theory includes interrelated concepts. The closer those concepts are along some scale of abstraction, the more potential the theory has as an insight-generation device. The traditional approach of focusing on the relationship between objects and concepts (in empirical research) may be necessary; however, by itself it is clearly insufficient to the task of developing more effective theories. We must use both if we are to build effective theories.
As a point of proof about the comparability of concepts, I must risk a recursion that is potentially amusing, thought-provoking, and informative. Very briefly, in this paper, I have presented a number of concepts for comparison. Because you, the reader, have compared them in your mind, you have already conducted the appropriate experiment.
It is worth mentioning, that each concept may be understood as an attribute variable. And, that each person has different models. Thus, by better understanding the interrelationships between those concepts, we gain new insights into how to support multidisciplinary integration research. Indeed, this kind of approach opens a new door for integrating theories within and between disciplines to impel the sciences toward an interdisciplinary direction. Following this approach, a hypothetical group of scholars should first seek to shift their individual theories toward the same level of abstraction before striving to link them. The resulting, more effectively integrated theories, would pave the way for more effective interdisciplinary collaboration and greater effectiveness in practical application.
An example may be found in comparing two conceptual systems from the physical sciences; one that is effective in practical application and one that is not. About 1600, Gilbert developed a theory of electrostatic attraction that included a diverse set of abstractions. These included very concrete concepts (e.g., “magnets”), more abstract concepts (e.g., “form”), and some (e.g., “attraction”) that may be somewhere in-between (Roller & Roller, 1954). That theory was not nearly as useful as the Coulomb’s conceptual developed in the following century which included “distance,” “force,” and “charge;” arguably existing closer to one another on some scale of abstraction. What we now know as Coulomb’s Law is far more useful than Gilbert’s conceptual system. Coulomb’s Law supports the design of electronic devices including cell phones and computers—far outstripping the usefulness of Gilbert’s version.
In the social sciences, it is difficult to compare ineffective conceptual systems with effective ones. There are many of the former and vanishingly few of the later. Organizational learning theory as an example has not been shown to be highly successful (Wallis, 2009a). There, the abstraction of concepts range from the very abstract (e.g., “formalization of knowledge”) to the rather concrete (e.g., “the creation of new firms).
A rare exception may be found in the comparison of two economic policies where one failed and one succeeded (Wallis, 2011). The failed policy (Prices and Income Accord—implemented in Australia in early 1980s) consists of a range of concepts from “centralized wage control” which may be seen as rather abstract (especially because it is contrasted in the policy with piecemeal wage control) to “spending” which is arguably more concrete. The more successful policy (Wassenaar Accord—implemented in the Netherlands about the same time) contained relatively few concepts (including economic recovery, competitiveness, price levels, and others). The Netherland’s version may be seen as existing at a similar level of abstraction compared to the Australian version. However, more studies are necessary to clarify and quantify the relationship between similarities of abstraction and success.
To summarize, this paper adopts a science of conceptual systems perspective and suggests that it is useful for developing a new understanding of theory. This applies to social and behavioral sciences such as business, psychology, sociology, economics, and policy. It may also apply to other fields and sciences. This paper first argued that a well-constructed theory should contain concepts that are of the same level of abstraction. For example, a theory should not contain one concept with a low level of abstraction (e.g., pippin apple), and another concept is more abstract (e.g., fruit). This implies that the internal validity of the theory will be enhanced to the extent that all the concepts are at the same level of abstraction. I know of no studies that have examined this relationship. So, it seems the gates are open for a new stream of research in this emerging science of conceptual systems.
Another opportunity for action involves the creation of a method to analyze and measure the levels of abstraction with some degree of objectivity. And, thereby, provide a new method for scholars and editors to evaluate the internal validity of a theory under construction or under submission to a journal. Such a methodology might reflect, for example, the percentage of all concepts within the theory that are at the same level of scale. That percentage, in turn, might be correlated with the scale of abstraction of the theory as a whole—as it relates to the world at large (unit level to grand level). This relationship would indicate the extent to which a theory remains true to its stated role. This kind of measurement might also be useful in creating a “periodic table of theories” as originally suggested by Feuer (1995). Here, I deepen and extend his idea by suggesting that such a table could include scale of abstraction along with other relevant features of the theory such as complexity and systemicity (Wallis, 2012).
This paper leaves open an important question about how we might build theories that work between levels of abstraction. The main suggestion I have in that regard is that we should develop a new understanding of theory—where theories at one level of scale (e.g., apples) are somehow nested within theories of another scale (e.g., fruit). The connection between levels, then, might be understood as an emergent property.
We might also seek to understand potential benefits of linking concepts at the very highest levels of abstraction, or the very lowest. And, we may seek to understand the benefits of creating conceptual systems with a greater number of concepts, fewer concepts, and those concepts which are linked through causal relationships (e.g., Wallis, 2013) to identify which may be more useful in practical application.
The above conversation around the scale of abstraction seems useful (and, potentially, important) because it provides scholars with a relatively objective approach to create more effective theory and evaluate our theory more effectively. Such theories are expected to be more useful and effective in practical application. By developing more rigorous approaches to understanding conceptual systems, we can learn to go beyond the empirical/positivist approach of Newton and leap forward to the poly-conceptual approach of Einstein.
Ackermann, F., & Eden, C. (2004). Using causal mapping - individual and group, traditional and new. In M. Pidd (ed.), Systems Modeling: Theory and Practice (pp. 127-143). Chichester, UK: john Wiley & Sons.
Ambrose, D. (1996). Unifying theories of creativity: Metaphorical thought and the unification process. New Ideas in Psychology, 14(3), 257-267.
Appelbaum, R. P. (1970). Theories of Social Change. Chicago: Markham.
Axelrod, R., & Cohen, M. D. (2000). Harnessing Complexity: Organizational Implications of a Scientific Frontier. New York: Basic Books.
Baake, K. (2003). Metaphor and Knowledge: The Challenges of Writing Science (Studies in Scientific and Technical Communication): State University of New York Press.
Bernier, L., & Hafsi, T. (2007). The changing nature of public entrepreneurship. Public Administration Review, 67(3), 488-503.
Boudon, R. (1986). Theories of Social Change (J. C. Whitehouse, Trans.). Cambridge, UK: Polity Press.
Burrell, G. (1997). Pandemonium: Towards a Retro-Organizational Theory. Thousand Oaks, California: Sage.
Curseu, P., Schalk, R., & Schruijer, S. (2010). The use of cognitive mapping in eliciting and evaluating group cognitions. Journal of Applied Social Psychology, 40(5), 1258-1291.
Daneke, G., A. (1999). Systemic Choices: Nonlinear Dynamics and Practical Management. Ann Arbor: The University of Michigan Press.
Daneke, G. A. (1997). From metaphor to method: Nonlinear science and practical management. International Journal of Organizational Analysis, 5(3), 249.
Dekkers (2008). Adapting organizations: The instance of Business process re-engineering. Systems Research and Behavioral Science, 25(1).
Dubin, R. (1978). Theory building (Revised ed.). New York: The Free Press.
Fedigan, L. (1973). Conceptual systems theory and teaching. Educational Leadership, 30(8), 765-968.
Feuer, L. S. (1995). Varieties of scientific experience: Emotive aims in scientific hypotheses. Piscataway, NJ: Transaction.
Fodor, J. A. (1998). Concepts: Where Cognitive Science Went Wrong (Oxford Cognitive Science). new York: Oxford University Press.
Friedman, K. (2003). Theory construction in design research: Criteria: Approaches, and methods. Design Studies, 24(6), 507-522.
Fuller, T., & Moran, P. (2000). Moving beyond metaphor. Emergence, 2(1), 50-71.
Ghoshal, S. (2005). Bad management theories are destroying good management practices. Academy of Management Learning & Education, 4(1), 75-91.
Greenhalgh, T., Robert, g., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science and Medicine, 61, 417-430.
Hammond, D. (2003). The Science of Synthesis: Exploring the Social Implications of General Systems Theory. Boulder, Colorado: University Press.
Hatch, M. J., & Yanow, D. (2008). Methodology by Metaphor: Ways of Seeing in Painting and Research. Organization Studies, 29(1), 23-44, doi:10.1177/0170840607086635.
Hung, D. W. L. (2002). Metaphorical ideas as mediating artifacts for the social construction of knowledge: Implications from the writings of Dewey and Vygotsky. International Journal of Instructional Media, 29(2), 197.
James, W. (1909). A Pluralistic Universe (Hibbert Lectures at Manchester College on the Present Situation in Philosophy). Manchester, UK.
Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral science (Chandler Publications in Anthropology and Sociology). San Francisco: Chandler Publishing Company.
Kessler, E. H. (2001). The idols of organizational theory from Francis Bacon to the Dilbert Principle. [Essay]. Journal of Management Inquiry, 10(4), 285-297.
Kuipers, B. (1982). The "map in the head" metaphor. Environment and Behavior, 14(2), 202-220.
Lakoff, G., & Johnson, M. (1980). Metaphors We Live By: University of Chicago Press.
Letiche, H., & van Uden, J. (1998). Answers to a Discussion Note: On the 'Metaphor of the Metaphor'. Organization Studies, 19(6), 1029-1033, doi:10.1177/017084069801900606.
MacIntosh, R., & MacLean, D. (1999). Conditioned emergence: A dissipative structures approach to transformation. Strategic Management Journal, 20(4), 297-316.
Morgan, G. (1980). Paradigms, Metaphors, and Puzzle Solving in Organization Theory. Administrative Science Quarterly, 25(4), 605-622.
Morgeson, F. P., & Hofmann, D. A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of Management Review, 24(2), 249-265.
Ostroff, C., & Bowen, D. E. (2000). Moving HR to a higher level: HR practices and organizational effectiveness. In K. J. Klein, & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 211-266). San Francisco: Jossey-Bass.
Phillips, B., & Johnstone, L. (2007). *The Invisible Crisis of Modern Sociology: Reconstructing Sociology's Fundamental. Boulder, Colorado: Paradigm.
Quine, W. V. O. (1980). From a logical point of view (2, Revised ed.). Cambridge, MA: Harvard University Press.
Robbins, D. (2000). Vygotsky's Psychology-Philosophy: A Metaphor for Language Theory and Learning. New York: Kluwer.
Rogers, P. J. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), 29.
Roller, D., & Roller, D. H. D. (1954). The Development of the Concept of Electric Charge: Electricity from the Greeks to Coulomb (Vol. 8, Harvard Case Histories in Experimental Science). Cambridge, Mass: Harvard University Press.
Seabury, M. B. (1991). Critical thinking via the abstraction ladder. English Journal, 80(2), 44-49.
Shotter, J. (2005). Inside the moment of managing: Wittgenstein and the everyday dynamics of our expressive-responsive activities. Organization Studies, 26(1), 113-135.
Smith, M. E. (2003). Changing an organization's culture: Correlates of success and failure. Leadership & Organization Development Journal, 24(5), 249-261.
Speicher, M., Marga (1997). Theory, metatheory, metaphor--introduction. Clinical social work journal, 25(1), 7-9.
Stacey, R. D. (1996). Complexity and Creativity in Organizations. San Francisco: Berrett-Koehler Publishers, Inc.
Tetlock, P. E. (1985). Integrative complexity of American and Soviet foreign policy rhetoric: A time-series analysis. Journal of Personality and Social Psychology, 49(6), 1565-1585.
Thagard, P., & Stewart, T. C. (2011). The AHA! experience: Creativity through emergent binding in neural networks. Cognitive Science, 35(1), 1-33.
Uzzi, B., & Spiro, J. (2005). Collaboration and creativity: The small world problem. American Journal of Sociology, 111(2), 447-504.
Wallis, S. E. (2008). Validation of theory: Exploring and reframing Popper's worlds. Integral Review, 4(2), 71-91.
Wallis, S. E. (2009a). Seeking the robust core of organizational learning theory. International Journal of Collaborative Enterprise, 1(2), 180-193.
Wallis, S. E. (2009b). Seeking the robust core of social entrepreneurship theory. In J. A. Goldstein, J. K. Hazy, & J. Silberstang (eds.), Social Entrepreneurship & Complexity. Litchfield Park, AZ: ISCE Publishing.
Wallis, S. E. (2010a). The structure of theory and the structure of scientific revolutions: What constitutes an advance in theory? In S. E. Wallis (ed.), Cybernetics and systems theory in management: Views, tools, and advancements (pp. 151-174). Hershey, PA: IGI Global.
Wallis, S. E. (2010b). Toward a science of metatheory. Integral Review, 6(Special Issue: "Emerging Perspectives of Metatheory and Theory").
Wallis, S. E. (2010c). Towards developing effective ethics for effective behavior. Social Responsibility Journal, 6(4), 536-550.
Wallis, S. E. (2011). Avoiding policy failure: A workable approach. Litchfield Park, AZ: Emergent Publications.
Wallis, S. E. (2012). A Systems Approach to Understanding Theory: Finding the Core, Identifying Opportunities for Improvement, and Integrating Fragmented Fields. Systems Research and Behavioral Science, 31(1): 23-31.
Wallis, S. E. (2013). How to choose between policy proposals: A simple tool based on systems thinking and complexity theory. Emergence: Complexity & Organization, 15(3), 94-120.