The first thing we can do is learn about the concept of essential complexity and accidental complexity. This concept was introduced by Turing Award winner Fred Brooks who is mostly known for his paper “No Silver Bullet—Essence and Accident in Software Engineering”. So let us have a look at it: (View Highlight)
Accidental complexity refers to the complexity that arises due to the implementation details of a system rather than its essential nature. In other words, it is the complexity that is not inherent in the problem being solved, but rather a result of the particular way the solution is constructed. Or in other words: It’s the stuff you don’t need to solve a problem! (View Highlight)
Over-engineering, where a system is designed with more complexity than is necessary, can lead to accidentally complex code. This can result in unnecessary features or functionality that increase the codebase’s complexity without providing any real benefit to the user. (View Highlight)
Many software systems rely on a large number of external libraries and dependencies. Managing these dependencies can become accidentally complex if they have many interdependencies or require specific versions, making it difficult to keep track of which dependencies are needed and which are not. Often is a result of over-engineering. (View Highlight)
The process of configuring a system can become accidentally complex if it requires a long list of settings to be adjusted, many of which are not well-documented or understood by the user and most of them may not be necessary. This can make it difficult to set up, test and maintain the system, and can also increase the risk of errors and inconsistencies. (View Highlight)
Complexity is not only a “technical” problem. Feature complexity is when you don’t know the value the feature has to a user. An untested (not gathering feedback from users) feature is always accidentally complex because you don’t know yet if it’s needed from the user’s perspective. (View Highlight)
When accidental complexity is the stuff you don’t need, essential complexity is what you cannot avoid.
It refers to the inherent complexity of a problem or task that cannot be eliminated through simplification or abstraction. It is the complexity that is inherent in the problem domain itself, rather than in the implementation of a solution (View Highlight)
A good example of essential complexity is cryptography. It involves the secure transmission of information and is inherently complex due to the mathematical principles and algorithms involved in encryption and decryption. While some aspects of cryptography can be simplified or abstracted, the fundamental complexity of secure information transmission cannot be eliminated. (View Highlight)
n software engineering, we are especially interested in behavioral complexity.
This type of complexity arises from the interactions between different parts of the system, as well as between the system and its users. The nature of behavioral complexity is the lack of predictability of how the system reacts to interaction or change (View Highlight)
Large and highly coupled code is unable to change easily without a high risk of introducing bugs for example. (View Highlight)
And there is a system that is tremendously complex that we all know well, even though it has nothing to do with computers, the human body. It is a complex system because it consists of many different parts, each of which has its unique structure, function, and behavior. These parts include organs, muscles, nerves, glands, hormones, etc. which work together in intricate ways to sustain life and maintain homeostasis. The interactions between different parts of the body are often nonlinear, meaning that small changes in one component can have significant effects on the behavior of the entire system. (View Highlight)
Ask yourself questions like: Do we need a dependency injection framework to fulfill the pattern? Do we need a cache right from the beginning, because we might need it later? Do we need to depend on this huge library even though we only use a tiny bit of functionality of it, that we could easily implement on our own? Do you need 8 layers of abstraction in your code because you blindly followed some methodology? And do you have to use all the fancy new language features that only bloat your code and deliver no real value rather than make only you happy? Mostly the answer is no. I say: “The best code is boring code”. (View Highlight)
Let’s say your system needs some kind of persistence mechanism to do its job. Maybe another to-do list app. Don’t start with the decision on what kind of database you should implement. In most cases, the type of db does not matter at the beginning. Start with what is essential to your system, the business rules or higher-order policies. Define an interface for the persistence logic. Introduce what Uncle Bob referred to as an architectural boundary in his book “Clean Architecture”. This makes the implementation a plugin. A detail that can be changed later. Start with the easiest and simplest implementation that fulfills your early needs. In most cases, this would be an in-memory db to use as a fake in your unit tests. Later on, you can add an implementation (plugin) for an SQL or NO-SQL db or whatever type of DB you need. Maybe a simple file storage system is sufficient enough. This prevents you from introducing accidental complexity from the beginning. (View Highlight)
So do we need to write new code to give users the capability to solve a problem? Or even worse, set up a whole new microservice before we even know what the problem is or in other words what kind of capabilities the user needs to solve his problems. We might use an existing API to at least give some capabilities to the users and then measure if our assumptions meet reality. Don’t build the stakeholder’s fancy tool to manage the new feature right from the beginning, before you even know if the feature will be successful. Use tools you already have, even if it does not fit perfectly. After you gained some more insights, re-evaluate the decision and move on which might mean writing new code. (View Highlight)
“Lean Experiments are based on the Lean Startup approach to creating new products and services under conditions of extreme uncertainty. Lean Experiments are designed to quickly and cheaply gather evidence to validate or invalidate risky assumptions about your product.” (View Highlight)
Hypothesis: Clearly state the hypothesis you want to test. For example: “If we add a feature that allows users to easily save their progress, then more users will complete the sign-up process.” (View Highlight)
Metrics: Define the metrics you will use to measure the success of the experiment. These should be tied to your hypothesis. The metrics should be a part of your hypothesis. For example: If we add a feature that allows users to easily save their progress, then 20% more users will complete the sign-up process.” Without a metric, your hypothesis is just an assumption. It is useless because it can not be tested and therefore not validated. (View Highlight)
Methodology: Describe how you will run the experiment. This should include details such as the sample size, the time frame for the experiment, and any control groups you will use. For example: “We will randomly assign half of our users to the experimental group, which will have the new feature. The other half will be in the control group, which will not have the new feature. We will run the experiment for one week and measure the success metrics.” (View Highlight)
Prototyping: Develop a prototype of the feature or change you want to test. This should be quick and inexpensive, focusing on the core functionality that you want to test. So instead of a Minimal Valuable Product define an even smaller Minimal Testable Product. But bear in mind that your results heavily depend on what you test. If you test for revenue, you need to have the whole behavior of the feature implemented. Most of the time you’ll end up implementing the MVP feature. But if you want to test if the majority of users would use the feature, a UI-only implementation with no real functionality might be just enough to gain first insides. (View Highlight)
Testing: Run the experiment according to your methodology, collecting data on your success metrics. Be sure to track any unexpected results or issues that arise during the experiment. (View Highlight)
Analysis: Analyze the data from the experiment, comparing the success metrics for the experimental group and the control group. Determine whether the hypothesis was supported or not. (View Highlight)
Next steps: Based on the results of the experiment, decide on the next steps. If the hypothesis was supported, consider implementing the feature or change more broadly. If the hypothesis was not supported, consider iterating on the prototype and running the experiment again, or pivoting to a different approach altogether. (View Highlight)
If complexity is the lack of knowledge of how a system behaves, especially when changes are made to it, we should find a coherent structure to separate the parts of our system that should not be interdependent. When we reduce the coupling between stuff that is not coherent, we only need to “load” the stuff in our brains that has the same concern, when applying changes to it and therefore reduce the possibility that changes affect too many other parts of the system that we are not able to track at once. It’s divide and conquer. The concern of a thing plays an important role here. But not only one. Make sure that the parts of your system are responsible only for one thing. But responsibility should be interpreted differently here. It´s about the actor or driver of change. This can be a group of users or stakeholder, but not only. Keep this also in mind when deciding which behavior should be grouped into one coherent module. (View Highlight)
The effort that is needed to make changes to your system is called the Cost of Change. When the product development cycles are short, the learnings come in quickly and the need for change also. Therefore, your system needs to be agile too. The cost of change must be as low as possible. We need to adapt the changes to the system quickly, because if you learn that you did something wrong, the faster you can make changes to your system and re-run the experiment, the faster you will make progress. (View Highlight)