RESEARCH - Normal Chaos

Introduction

 

These pages are for those kinds people who have offered to support the research needed in order to write my third book.  This book will be looking at the implications for management when we accept normal chaos as our basic paradigm rather than staying within the perfect world paradigm. I will expand these pages as our thoughts develop. Please note that phrases in italics are technical terms derived from academic literature and therefore imply much more than the words, in themselves, convey.

 

In all that you might read in the following pages that, while complex, offer great opportunities, we look at the issues from the perspective of how they may disrupt our plans. We are asking how complexity may create risk and be the source of the next crisis. This concern fundamentally shapes and biases all our thinking, an approach we call “constructive pessimism”; pessimism because we are always thinking about the unwanted occurrence that may happen and constructive because this, in our view, gives us a better chance of preventing or, at least, mitigating it.

 

What is the point of this Research?

 

In brief, what I am researching is exploring a new way of looking at issues in order to promote foresight. To me the essence of foresight is being able to prevent failures or crisis and, if we cannot prevent them, then ensuring that we are prepared as well as we can be to manage them. In this context I see foresight being delivered by what Weick and Sutcliffe call Mindfulness. In more straightforward language this is about being aware what is actually happening to us and around us from moment to moment. I have already written about why we might not see the world as it actually is, now I am looking at ways of helping us see the reality of what we face. The key underlying assumption of this work is that the better we understand the world the more likely it is that we can predict how events will develop; in the meantime the best we can do is muddle through on imperfect information using imperfect systems.

 

However, foresight requires more than just being in the moment. It also requires the practitioner to grapple with the issue of emergence.  Emergence concerns where a seemingly stable system throws out an unusual result. Emergence concerns situations where changes, even very small and apparently insignificant changes, can produce significantly different results. Now we are in the realm of needing to understand the techniques of Tetlock and Gardners’s super-forecasters (who can achieve up to 60% accuracy) and the hacker mindset as described by Tim Summers. Summers describes how "Skilled hackers perform at the edge of the unknown within poorly defined domains (where they have to) leverage knowledge, creativity, curiosity, expertise, and interpretive schemes.” Summers describes how hackers face problems  that are “poorly defined, ever-changing domains with many peculiar questions and issues that lack clarity and specification"; I would suggest that this is also a good description of the normal ever day working environment of a business executive! For these executives the question must be one that is concerned with how well they leverage their knowledge, creativity, curiosity, expertise, and interpretive schemes.

 

So, I see the idea of normal chaos as a way to enhance foresight as it is a construct that is aligned with mindfulness, super-forecasting and the “hacker’s mindset”. The work we are currently doing is designed to test this proposition.

 

Developing Thoughts

 

This research builds on my previous work that focused on whether it is realistic to assume that all failures of foresight can be avoided.  This has led me to the conclusion that this ambitious goal is unrealistic. This belief comes from the fact that the world we live in is so complex that unexpected outcomes appear from apparently normal interactions, that many of our actions have unintended consequences and factors that we might assume to be stable and consistent are not, they fluctuate  - and only if we are lucky - within set parameters. Given these factors, patterns of activities that we perceive may be illusory and where, in many cases, we cannot even perceive there being a pattern, might therefore justify our seeing the world around us as being chaotic. Here the term chaotic is as used in chaos theory (undetermined patterns) rather than in the common usage sense of "complete disorder and confusion". I have described my rationale for believing in the Normal Chaos paradigm separately (See Coping with Chaos)

 

Within the context of Normal Chaos, while we may still be able to create within the chaos small oases of order, these can vanish in a moment; hence this is why even high reliability organisations can suffer failures. As humans we find it difficult to cope with the uncertainty this chaos brings and so, in order to create "understanding", we create patterns where there may be none. These coping mechanisms range from blame and denial to religion. These mechanisms can be seen at work in the way inquiries by the judiciary or the press often reduce complex problems down to placing blame on those nearest to events; this habit is like blaming the driver of the last truck over a bridge before its collapse. (If you are interested in the detail you can read more about this phenomenon within the literature dedicated to illusions of control.) As I examine management systems I see that many management tools embrace these same delusions of control and stability. This has led me to examine the implications for management if we accept normal chaos as our working paradigm.

Early on in my research a colleague sent me a photography that neatly summed up the difference between a Perfect World and Normal Chaos ...

 

The idea at the heart of my current work is to examine how in our working lives our current management mechanisms and tools enable or impede us from coping with the world we face.

 

My starting point will be to contrast the ideas of the perfect world paradigm and normal chaos. I first introduced these terms in my second book. I have since expanded my thinking on both of these paradigms. As I will be using this research to develop these ideas further, I will not expand on them here; these ideas are not yet stable enough to present in a structured form. They will however be a key feature of all future discussions.  

 

In general terms then, rather than using the perfect world paradigm, I have championed the use of the paradigm of normal chaos. Here the key idea is that the patterns of life that surround us all are too complex for the human mind to grasp; we therefore need to simplify the world we see in order to make sense of it which leaves us open to missing vital details that can have significant effect upon us… So how do we cope in reality? 

 

Coping With Chaos

[Last amended: 29 Nov 21]

 

How do we Cope when Chaos is the Norm?

 

Basic Assumption

 

Basic assumptions at this point are:

 

  • that those engaged in this research already accept that Normal Chaos rather than the Perfect World is the accepted paradigm.

  • the purpose of the plans or rule-set is clear and agreed by those involved.

 

[What constitutes purpose may be more complex than we originally imagined. Dee Hock (the doyenne of chaordic organisations) has stated that "a purpose is not an objective, it's not a mission statement—a purpose is an unambiguous expression of that which people jointly wish to become." Initially we will therefore take an organisation's purpose as a given but as we review the principles we need to stay alert to the need to amend the given purpose.]

 

Background

 

The world that confronts us is ever changing; to use a truism that has become a cliché, "the only certainty is uncertainty". Our ability to forecast the future has proven wholly inadequate and yet, the methods we use to run our lives and our organisations are based on creating certainty. This tactic manifests itself in the plans and rule sets that we use in an effort to control the world around us. Experience shows that such an approach is flawed.

 

Plans often fail to deliver the desired result and rules fail to provide a fool-proof way of avoiding the crisis or disaster. There is a general rule that the more senior the manager, the more time he or she spends resolving conflicts that arise due to the complex interaction taking place within their organisation. In these cases, the manager is forced to makes choices that constitute the least-worst rather than the best option.

 

My study of organisational failures of foresight has highlighted the almost universal issue that the world the organisation has designed itself to face is not the world that confronts it; what actually occurs exceeds the range of our expectations and the variations envisaged. This is what we refer to as Normal Chaos.

 

Our work to date has enabled us to identify there being some significant implications of accepting the Normal Chaos paradigm. These implications, that offer eleven assumptions about the world in which we live and work, were identified by a Delphic discourse of those involved. The assumptions are:

 

1. The world we live and work in is complex and driven by forces that we often do not see, recognise or appreciate

2. All actions we take have consequences, and these actions have both upsides and downsides whether they are obvious or not.

 

3. We should always expect the unexpected and consider where unintended consequences of our actions might emerge.

 

4. We live in a world of continuous change that thwarts our plans; therefore we are constantly forced to adapt the plans.

 

5. Given that "no plan survives contact with the enemy", we should see that having an effective planning process is more important than having a good plan.

 

6. Due to the world's interactive complexity, our understanding of the problems we face will always be only partial.

 

7. The patterns we see in the world around us are often temporary, dependent on the context and the scale of observation. Hence, these patterns may simply be illusory. That’s why we need to be cautious about basing our plans on them.

 

8. There are no “universal solutions” to problems! All solutions are contingent on the circumstances to which they are to be applied.

 

9. Management will always require a mix of “craft” (‘intuitive skills’) alongside compliance with laws and regulations to cope with the prevailing uncertainty that surrounds them.

 

10. Our ability to control what happens to us and our organisation is much more limited than is normally assumed.

 

11. Success is always relative rather than absolute.

 

In 2018 we produced our first draft of a catalytic framework based on the ideas contained within chaos and complexity theory. Through an inductive process we have identified three categories of properties (Structure, Patterns and Energy) that we consider as having potential qualitative use (to stimulate discourse) in the context of normal chaos. Each category is seen as being made up of three parts, these parts being selected from: systems scope, interdependencies, self-organising, illusions of stability, fractals, fitness landscape, attractors, energy flow and the edge  of chaos. Each of these ideas embraces a rich mix of ideas and concepts that can be used to provoke a greater understanding of the issue and context of concern.

 

For ease of conceptualisation the ideas are laid out as a network of ideas. However, it is important to remember that every item interacts with the others so that linkages are much more elaborate than can be set out clearly in only a two dimensional format such as the one we have here.

 

Since producing the original framework it has undergone a number of revisions in order the try to clarify the ideas being conveyed.

web-NormalChaos

A fuller description of the components of this framework can be found here.

 

Aligning the Prefect World Paradigm with Normal Chaos

 

 

Ever since I first used the term "Normal Chaos" in my 2013 book "It Should Never Happen Again", I have been trying to refine what I mean by it. In my 2015 book "In Pursuit of Foresight" I expanded my thinking further, juxtaposing the idea of Normal Chaos with that of the Perfect World paradigm. This practice of setting one idea against an alternative is common within the academic world where their arguments are based on the fact that two competing ideas cannot both be correct. In my early research I explored the debate between the merits of High Reliability Organisations and, what is referred to as, Normal Accident theory. As a former practitioner, I could see merits and flaws in both constructs. However, within the academic literature each researcher seemed to believe that one was right and the other was wrong. I was surprised how bitter this discussion became! In the end I wrote to Charles Perrow asking him to explain his intent in writing about Normal Accidents. He replied that he was not theorising but warning that complex closely coupled systems will inevitably go wrong and, if the consequence of that failure was significant, we should think again before we commission such systems. My exchange with Prof Perrow made me start to think about complexity as a factor in organisational failures.

 

Despite this previous scholarship, when it came to my exploration of Normal Chaos and what I then saw as its antithesis, the Perfect World paradigm, I still approached my research on the flawed basis that if one was true then the other must be false. I have now changed my mind. I now see one (the Perfect World paradigm) as being a subset of the other. Let me explain.

 

The basic proposition of the Perfect World paradigm is that if we recruit the perfect people, produce perfect plans, train them perfectly, supply them with exactly the right resources (including perfect unambiguous information) and execute the resulting plan flawlessly (eliminating all slips and lapses) then the desired outcome will be delivered. Within this paradigm is the belief that individuals should be able to learn, retain and use the knowledge they require perfectly. All of this perfection is then supported by having perfect foresight leading to individuals being blamed and punished where they fail to achieve these standards. Embedded within this construct is the desire to remove uncertainty and to control the world around us. The label Perfect World paradigm is used to reflect the phrase often heard when discussing failure; that is “but in a perfect world …”. At this point we need to ask the question as to whether this perfect paradigm could ever hold true. My answer would be both "yes" and "no"! I see the dividing line between yes and no coming down to the granularity (in Normal Chaos terminology "scale") of the criteria used to judge perfection and the paradigm’s practical utility.

 

The basic premise behind the Perfect World paradigm is that you can produce closed systems and operate within them. That is a system that can be isolated from external disruption. This assumption can be seen in operation across many practices ranging from the micro to the macro.   Rule-based systems are an example of the paradigm working at the micro level. At the mezzo level we can see examples such as the functioning of a factory, and at the macro level, we might cite international just-in-time logistic processes. All these systems are basically linear in nature, their inputs are thought of as being predictable and these processes are set within clear boundaries.  Whether the answer to my question is yes or no will therefore depend on the scale of variance that can still be considered to be operating perfectly, against an absolute standard set for the system. In practice this would be judged on whether the system or processes broke down or not. In practical terms the idea of perfection is quite elastic!

 

To give the answer "yes", would be to accept the minor fluctuations, slips and errors that are inevitable within the system. Experienced managers will recognise how much of their working day is absorbed by deciding on what fluctuations within their system are acceptable or the action required to ensure the process is brought back to within acceptable boundaries. Within the terminology that I used, this would be seen as being a self-regulating system. The criteria for what is acceptable generally comes down to whether the product or service continues to be delivered to the client in a manner that is acceptable to all stakeholders that the key parties care about [who these stakeholders might be will be the subject of a future blog]. The utilisation of the Perfect World paradigm is therefore seen to be useful to practitioners when they feel that they can control all the key parameters within their system. The linear nature of this paradigm also makes it easier to apply. The test for such systems is that they are rule-based and, in general, the rules remain valid and can be applied.

 

Use of the Perfect World paradigm starts to fail when, in practice, managers find that they do not have as much control over their system as they thought or hoped they had. They see their rule-based systems (including its contingency options) are not able to cope with the variance encountered. This is the point my "yes" answer would turn to "no". In practice (and in order to ensure the continuing delivery of a product or service) the use of this paradigm fails when an input to the system causes a major problem that requires a reconfiguration of the existing system, or the necessity for the rules (as written) to be violated. Within the terminology that I used, this would be seen as being a self-organising system. In this context the Perfect World paradigm continues to have practical utility up to the point that self-organisation and rule violations become the daily norm. In practice however, as the Perfect World paradigm is the predominant analytical lens taught at business schools and other centres of learning, managers often have to continue to use this paradigm way past its usefulness as they have no other means of assessing their management problems.

 

In practice, the point that self-organisation and rule violation becomes the daily norm is the point at which operations are likely to be recognised as being complex, dynamic and often, high tempo. In this context things happen quickly, and the patterns that we base our decision-making process around are more difficult to discern. In these circumstances, inexperienced practitioners are likely to see chaos (disorder). The more experienced practitioner would however be able to see the more complex patterns at work, including seeing what is not present that should be. They see order in the disorder; now we are approaching the ideas of Complexity, and the related Chaos, Theories. (I will elaborate my use of these theories separately). These theories consider open systems that function in non-linear ways. The open nature of these systems means that they can be expected to be subjected to many uncontrolled factors in what might appear to be a random manner.

 

The non-linear nature of these systems is likely to mean that many of the cause-and-effect relationships that exist within the system are difficult to determine. In practical terms these characteristics will be apparent from the number of external influences that disrupt the working routine. As these become more frequent, the rules that govern such systems will become increasingly complex and are frequently unworkable without violation. [It is worth noting that the production of rules or guidance in these circumstances is a very interesting stream within the academic literature on safety. From this work there is clear evidence that the application of rule-based systems as envisaged by many public inquiries is not appropriate for complex, dynamic and high tempo situations.]

 

Going back to the original question as to whether the Perfect World paradigm could ever hold true, the answer (from the point of view of scientific utility), would also be “no”. There is an old Yiddish adage that says, “Man Plans, and God Laughs”. This is taken to mean that despite our efforts, our plans are often wrecked by the unpredictable nature of life. In the more colourful language of the military, “shit happens”!  The probability of internal missteps and unwanted external influences mean that, unless the scope is very restricted or the planner is incredibly lucky, it is unlikely that any complex plan will ever be successfully enacted without adjustment. The irony is that the imperfection of the Perfect World paradigm cannot just be ignored.  This means that, of the two ways of seeing the world, only Normal Chaos has scientific validity. However, as it does have practical utility, to reject the Perfect World from this debate would also, in my view, be wrong. 

 

So, what is Normal Chaos? It is the recognition that the world in which we operate is an open system where all our plans are liable to disruption by both internal and external factors. It recognises that, while we may try to isolate our systems and processes from external factors, they are always liable to the effect and disruption of such influences. The Nobel winning physicist Ilya Prigogine called these spaces "islands of order in a sea of disorder". Normal Chaos recognises that as our systems and processes become more complex, they are likely to produce unexpected results; they produce patterns of activity that the inexperienced are likely to see as being chaotic (disordered) while the more experienced practitioners are more likely to see the patterns in play. One of the features of such systems is that they produce a regular outcome multiple times through the mechanism of dynamic stability: these repeated results can be mistaken for the system being stable. This means that when they unexpectedly produce a different result, this takes the people affected by surprise: these are often seen as Black Swan events while in fact that are just emergent events to which people have been blinded by the induction fallacy. And so, we start a new way of thinking.

 

The implication for practitioners of accepting the Normal Chaos paradigm is that they are always aware that nothing in their process or system actually stays the same and therefore they recognise that it is always in a state of flux that can produce apparently anomalous results. They recognise that while such a system may appear to be stable in the short-term, it is a dynamic stability that can change at any moment and therefore, in the long-term, it is not. Here one of the key judgements that practitioners have to make is an assessment of how long the current apparent status quo is likely to last, for this will give their deliberation structure. They need to assess the forces acting on those structures and to be prepared to adapt at the appropriate time. In order to plan their adaption strategy they will need to be alert to the way the relationships (seen as patterns) between the structures may change and how the structures might evolve in the longer-term. They have recognised that even small fluctuations can have dramatic effects on the results their system produces. As a consequence of this, those who embrace the concept of Normal Chaos are aware that their system might falter at any moment and therefore focus more on effectiveness, robustness, resilience and agility rather than producing the most efficient system possible.

 

Last updated: 14 Nov 21