For many, software engineering seems like mostly focusing on bits and bytes, disconnected and isolated from the hassles of real life. One can sit in front of the screen for hours, trying to solve a problem in an appropriate way..until the product manager comes and asks something “very urgent” out of nowhere…Or until that lucrative contract falls..or until..

Or may be, software engineering is not isolated from the hassles of real life at all. May be it lives right in the center of it, impacted by even wing flaps of a small butterfly.

Brewing their experiences over decades in software engineering, the thought leaders of our industry authored Agile Manifesto to present set of values and principles which gives organizations a better chance to stay relevant to their customers with sustainable quality in a fluctuating environment. Many considers the combination of Lean Software Development and Agile are our best chance of survival. It is not perfect, but we do not have a better alternative yet.

But why has Agile potential to work better than other approaches even when it is executed less than properly ?

In this post, I will try to answer this question with help of Nassim Nicholas Taleb’s infamous The Black Swan theory.

The Black Swan Theory

If you have not heard of N.N.Taleb before, you should stop and take a look here Shamelessly copying from Wikipedia:

Nassim Nicholas Talebis a Lebanese-American (of Antiochian Greek descent) essayist, scholar, mathematical statistician, and former option trader and risk analyst, whose work concerns problems of randomness, probability, and uncertainty. His 2007 book The Black Swan has been described by The Sunday Times as one of the twelve most influential books since World War II.

Black swan is a metaphor for events that are totally surprising from the perspective of the observer (but not necessarily from the perspective of the originator), have major impact and often rationalized after it occured as if it could be well predicted. Those are main characteristics.

Some of the black swan events Taleb gives as an example are the invention of the internet, personal computer, 9/11, World War 1, or collapse of the Soviet Union.

In his work, furthermore Taleb divides the reality in to two lands: mediocristan and extremistan.

Mediocristan is the land where we have a lot variations without none of them being extreme. Evolution belongs to mediocristan, tons of genetic mutations happening all the time, but nobody suddenly develops wings. In mediocristan, variations fit in to normal distribution. In mediocristan, no event can have enough impact to change the distribution substantially. Most of the time we think and act as if we live in mediocristan.

Extremistan on the other hand has low variations with some of them being extreme (Fat tails). Therefore the impact of a black swan event is consequential.

The world that we as humans created, business, economy, life, operates in extremistan. Many of our problems caused by our belief that we live in mediocristan and so we can have safe assumptions, forecasts, precise plans on stuff. That, is not our reality. So I would say:

Impact of Black Swans in Software Engineering

Now theory shows its marvel on large scale, large impact events. But I try and see if I can scale it down to an single industry, company, bunch of people and check if principles apply to much smaller areas.

I hope I do not make a mistake and attract the fury of N.N.Taleb His works trigger many thoughts and ideas everywhere.

The area of impact is a single company, or team rather than whole world, societies or countries. There are many events that can be consequential for the team or the company. In the worst case, the company may go bankrupt, team can be disbanded.

Now lets imagine some events that heavily impact our engineering efforts:

• An important customer requesting an immediate feature (surprise for us as observer, well known by the customer)
• A new regulation in a country we ended up having customers (surpise for us, not for the authors)
• That big customer who pays majority of the bills, going out of business..or changing its partner..or something else (Again, not surprise for the customer)
• Drastic changes in roadmaps due to fluctuating business requirements
• Serious event in the world shaking the businesses from its core (Political instabilities, economic issues, pandemics)

Could we predict any of those as a member of or company ? Probably no. If those are perfectly predictable, then we can proactively engage them before they happen. That level of predictability for such events contradicts the fact that we live in extremistan.

But this is not the case. We indeed must accept that we do not know what do not know. Moreover, we do not have control on events like that. We can only survive through them, gaining some advantage if we can play it properly.

Agile, embraces the unknowns exactly like this. It does not deny the fact that we can’t possibly know and predict everything. While waterfall based approaches treat the world like almost like a controlled environment with a strong self-esteem and long term detailed plans; a lean/agile mind inherently knows that the map is not the territory and positions itself to survive through consequential events.

Therefore an agile mind is closer to how the software world actually works.

Containing the Damage

What can we do when such a consequential event happens ?

We can either pick a more flexible, opportunistic, customer oriented approach like Lean/Agile, or something else that sits on the opposite side of the bar.

The cost of containing damage can skyrocket, if we follow a path that is inherently rigid and in denial of realities.

• Reduce risks as feasible, appropriate as possible, to contain the damage when the risk realizes.
• Be prepared and have the flexibility to make course corrections, to react on events without losing time

Both of those points map to the many of the Lean/Agile principles.

Conclusion

When we read The Black Swan, we are exposed to a clear portrayal of the world as it brutally exists. It is definitely not a recipe book. It teaches us how to look at the world in a different, much better way.

On the other hand, agile principles start with the assumption that our problem domain is complex, non-deterministic, empirical and continue by offering itself as a better sWe often do not question this assumption. But I believe, internalizing this assumption is important to see why those principles work better. If we dig in to failed agile transformations, more often than not, we will see that the organization still tried to deal with the world as if it was living in mediocristan.

If the world and the business we live in is part of the extremistan, it also implies that there will be some black swan events. If those events satisfy the criteria and happen consequential enough for our existence (team, company, product etc) we have to deal with them somehow. Since Lean/Agile principles naturally prepare us better than other approaches to deal with such events, we can say that Lean/Agile is a very good way to survive in extremistan.