I have become increasingly interested in understanding why it is so common for softwares projects to end up in the famous “Big Ball of Mud”, a state in which the incidental complexity of a system reaches a point that makes every small modification extremely painful.
Only once we understand the reasons we can design strategies to counteract this tendency. Among the many factors that play a role, I want to highlight one in particular in this post: There is an evolutionary pressure that causes hacky solutions to survive while well-designed software architectures tend to die out.
Fitness in the sense of the study of natural selection is not a normative concept. A trait that survives across the millennia might have given a species a slight edge when it comes to survival and reproduction in their environment – that does not automatically make the same trait morally or otherwise desirable.
Transferring the concept of natural selection to software projects is of course a flawed comparison, however, I am observing that the environment in which software projects are placed does favor some practices while others tend to “go extinct”.
The selection principle in the business world is usually economic success. Once a product delivers tangible value and you have people relying on your solution, a software product becomes hard to kill.
Projects that have not yet proven their value on the other hand find themselves exposed to mother nature’s whims which come in the form of political battles, strategic repositioning, or economic necessity. As long as value only exists in theory or on paper, you can always make an equally good argument why an initiative will fail.
Now, consider a solution that has been hacked together in a couple of days and that consists of a couple of Excel sheets glued together by some untested VBA code. You may cringe at the idea. However, the survival chances of this monstrosity are quite good: It has likely been produced by someone whose primary occupation is not building elegant systems but who found themselves faced with concrete, unresolved needs. They built something that addressed a concrete problem right away (requiring a very small investment).
On the other hand, imagine an engineering initiative designed to solve the problem once and for all: It uses all the right abstractions and best practices. Its implementation requires a team of five for at least half a year. The team produces tooling, tests, and documentation. This project can be killed at any moment depending on the strategy du jour and which management book the CEO read recently. Once that happens, likely only the people that invested time will be heard mourning its demise.
In the meantime, the Excel-based solution receives concrete feedback from its users, is extended, and grows in complexity. At some point, it will have grown to an extent that replacing it would come at a significant cost. Any move to kill it will face significant resistance from its user base. Real money is on the line!
Now, of course, there are strategies to escape extinction. The premise of this article is one of the main reasons the Lean movement exists. Nevertheless, it is useful to start with the humble realization that the deck is stacked against clean code and well-designed abstractions as neither of the two directly favors a project’s chances of survival.
The Solution Space 