How to systematically innovate from first principles to design a new process (Part 3)
Innovation is often a process that happens by chance, but there is a bottom-up approach to systematically innovating that we will cover together. In the last two articles, we discussed the problem of software getting rewritten and how to innovate using first principles. We will tie that all together and show you how to innovate systematically.
We will solve the problem of code bases getting rewritten, which costs considerable time and money. It seems like an enormous challenge, but with the help of first-principles thinking and this process, we will reduce something that seems inconceivably challenging into something manageable.
- Step 1. Understand your problems and/or goals and state them clearly
- Step 2. Invert the problem
- Step 3. Find our foundational principles
- Step 4. Be explicit about the constraints created by the principles - as applied to the problem - then combine them to come up with a solution
- Step 5. Coalesce, and iterate
Step 1. Understand your problems and/or goals and state them clearly
Understand our problems or goals and state them clearly.
To systematically innovate, we need to understand the problems or our goals. In simple terms, starting with something you don't want or something you do.
The problem of large code bases being rewritten is based upon a problem covered in a previous article. If you haven't read the article, you don't need to, as we'll cover the important points here, but it's a good partner article to this one.
So why are "Large code bases often rewritten?". In the previous post we discovered that the three root causes of rewriting large applications are as follows:
- Understanding is decreasing as complexity increases. People reach their cognitive limits, first as individuals and then as a team.
- The number of potential problem solvers diminishes as complexity increases, so the team has access to less cognitive power to solve problems as the system is less visible to everyone.
- The system becomes harder to change as the code base grows; it cannot evolve easily as it takes on a more rigid form. This rigidity prevents adaptation to future change. Big mistakes may already be baked in.
NOTE: Finding the root cause of a problem is not covered in this article but is critical to finding solutions and cannot be skipped. Use a root cause analysis (RCA) tool like the Many Whys AI-assisted app I created to help find root causes.
Now we've stated the problems, we're ready for the next step: inversion.
Step 2. Invert the problem
The problems we don't want -> inversion -> the things we do want
With the three problems stated, we need to invert them before matching them to principles. Inversion of problems is just the process of changing problems to goals - from what we don't want to what we do want. We need to do this because it opens up and allows us to see more possibilities, which helps find principles in the next step - As principles both work by exclusion (behaviours to avoid) or inclusion (behaviours to embrace).
Example
| Problem | Inversion |
|---|---|
| Understanding is decreasing as complexity increases. People reach their cognitive limits, first as individuals and then as a team. | We need a method that allows the code base to remain within the limits of the mind, to keep it understandable no matter how big it gets - We need transparent and relevant understanding. |
| The number of potential problem solvers diminishes as complexity increases, so the team has access to less cognitive power to solve problems as the system is less visible to everyone. | We need a way for many people to be able to help with a problem if they can help. We need collaborative problem-solving. |
| The system becomes harder to change as the code base grows; it cannot evolve easily as it takes on a more rigid form. This rigidity prevents adaptation to future change. Big mistakes may already be baked in. | We must ensure the system can rapidly evolve, even in large code bases. We need to prevent mistakes from becoming baked in. |
Step 3. Find our foundational principles
State what you want (or want to avoid) -> Create or find a principle that gets you what you want.
Now we have our problems and goals, we need to find the behaviours that can help us achieve what we want or avoid what we don't want. This is the most important step in the building blocks of first-principles thinking, it can also be the most time-consuming.
There are two ways to approach this process:
Option 1) (easier)
Find existing principles: Ones that can solve your problem, and use them to add to your behaviour.
Find other behaviours that would solve your problems and use them. Taking what others have already figured out and using it for ourselves is an efficient and rapid way of learning. Using others' principles is the best way to do that. We can use principles from this website or elsewhere.
Option 2) (hard)
Create a principle: Be explicit about the desired behaviour to solve your problem, then attach a strong rationale to it.
Figure out what behaviour would solve your issue and rationalise it. You might think this is a backwards approach; how can you rationalise towards a behaviour without first understanding the logic? But this is the discovery process, you have to have an aim otherwise, you'll go nowhere.
The best way is to write your own principle, because in the process of writing - which is just a sophisticated form of thinking - you will begin to put your assumptions to the test. You will nearly always find your original ideas were wrong, incomplete, or the behaviour you aim for is made up of multiple behaviours - multiple principles. There's no shortcut to this insight process, but whilst it can be hard, it is extremely valuable to your personal development.
Example
| What you want (or want to avoid) | Principle |
|---|---|
| transparent and relevant understanding. A principle that allows the code base to remain within the limits of the mind, to keep it understandable no matter how big it gets. | Make the invisible, visible states: "Make the invisible, visible" means: make relevant information visible so we can understand things clearly. |
| A system that can continually evolve, even in large code bases, ensuring that mistakes don't get baked in and ensuring the quality of evolvability. | The principle of evolvability states: Systems which can evolve quicker are better than those which cannot. Systems which have the quality of evolvability can rapidly adapt to their environment and change direction quickly, without being encumbered by previous evolutions. It allows the maximum amount of change with the minimum amount of effort. |
| To have multiple problem-solvers, not just engineers, that can work on high-level system problems collaboratively, not just 1 person. | Many minds, better problem-solving principle states: The greater the number of people who can work on a problem, the higher the probability of an optimal solution. |
Ok, now we have our principles that can combine together to create a new behaviour.
If you want to see how that works, look at the graphics in the previous article first principles thinking - a visual guide
Step 4. Be explicit about the constraints created by the principles - as applied to the problem - then combine them to come up with a solution
a. Take the problems -> Run it through the principles -> Write out the constraints
b. Combine all those constraints to come up with a solution.
Okay - so we've got our principles; we need to bring back our problems (or goals) and apply them to see their effect.
Example
a. Take the problems/goals -> Run it through the principles -> See the outputs.
i. Problems and goals:
| Problem | Problem Inverted |
|---|---|
| Understanding is decreasing as complexity increases. People reach their cognitive limits, first as individuals and then as a team. . | We need a method that allows the code base to remain within the limits of the mind, to keep it understandable no matter how big it gets |
| The number of potential problem solvers diminishes as complexity increases , so the team has access to less cognitive power to solve problems as the system is less visible to everyone. | We need a way for many people to be able to help with a problem if they can help. We need collaborative problem-solving. |
| The system becomes harder to change as the code base grows ; it cannot evolve easily as it takes on a more rigid form. This rigidity prevents adaptation to future change. Big mistakes may already be baked in. | We must ensure the system can rapidly evolve, even in large code bases. We need it to ensure it has the quality of evolvability . |
ii. Run it through the principles -> See the outputs.
Next we take all the inputs and run them through the logic specified in the princple to come up with one or more outputs.
| Principles | Outputs |
|---|---|
| Make the invisible, visible. | 1. We need a way to keep the system's relevant details understandable, regardless of size. No one will be able to understand all the details, but we don't need to. We need to find the correct level of abstraction so everyone on the team can understand things clearly. |
| 2. We need to prevent people from reaching their cognitive limits by only making visible relevant details. We need to reduce the resolution of details so any mind can understand the high-level details, similar to a map that gives you an overview of the territory but isn't the exact territory. | |
| Principle of evolvability | 1. We need a system that can be easily changed. The lower the friction the better. Code is a high-friction way of changing things because it needs to be understood, and it takes a long time to try and do things in a new way. We need a faster way to evolve the system without code. In fact the code shouldn't be the single source of truth, because it is not a great way to evolve the system, and if the code is the sole source of truth, it will inevitably cause everything else to rot. **We need to be able to design the system without code, a diagram that is abstract enough to provide value and can be changed easily. |
| Many minds, better problem-solving | 1. It would be ideal if problems could be shared cross-discipline. We know there are better mediums than code for this because only expert programmers understand the code. We need a medium that isn't code for sharing, a diagram of some sort that everyone can work on to solve problems. |
| 2. We need to prevent people from reaching their cognitive limits by only making visible relevant details. We need to reduce the resolution of details so any mind can understand the high-level details, similar to a map that gives you an overview of the territory but isn't the exact territory. |
The solution
- Combine all those constraints to come up with a solution.
Our solution must be:
- transparent and relevant understanding
- collaborative problem-solving
- quality of evolvability
- find the correct level of abstraction so everyone on the team can understand things clearly
- a map that gives you an overview of the territory but isn't the exact territory.
- the code shouldn't be the single source of truth
- We need to be able to design the system without code, a diagram which is abstract enough to provide value and can be changed easily.
- We need a medium that isn't code for sharing, a diagram of some sort that everyone can work on to solve problems
- We need a way of bringing the code and business together, something like ubiquitous language, from domain-driven design.
Combining all these, we can now synthesise our solution:
The map
We should create a map of the system (A medium that isn't code for sharing, a diagram of some sort) ( a map that gives you an overview of the territory but isn't the exact territory.) that shows the high-level logic of the system. The scale that we'll need to use for the map - the level of abstraction of the map should be based on - is at the business level, which means we'll model the business logic on the map. This is perfect because logic written in business terms will always be within the cognitive complexity of the team (find the correct level of abstraction so everyone on the team can understand things clearly). This is because business rules have an exciting property: they can never be more complex than is understandable; otherwise, they would be unimplementable in the first place. In contrast, a code base can be more complex than is understandable.
The map needs to be editable in real-time (a diagram of some sort that everyone can work on to solve problems.). An editable map allows everyone to see the system and solve problems together and even has the massive benefit of changing the system in a low-friction way - as it's just diagrams (quality of evolvability). As a map of the system from a business perspective, it is also a source of truth for our overall design from a business perspective (the code shouldn't be the single source of truth).
The code
To support the map and aid communication between the implementers (engineers) and everyone else, we should write code in a way that it solves business problems rather than just technical ones (We need a way of bringing the code and business together, something like ubiquitous language, from domain-driven design.). This avoids the translation that often happens between an engineering team and the business, resulting in the engineers having to understand two worlds: the business domain and a completely different technical implementation. The translation between these two worlds creates a lot of complexity in a code base. The system would remain significantly more understandable if the code had a layer that was written from a business perspective. Domain-driven design provides both ubiquitous language for modelling and communicating business realities. Clean architecture talks about a "use cases." layer, which is a layer that contains business rules. Use cases or other similar patterns that implement code in a way that uses business rules are ideal.
Step 5. Coalesce, and iterate
We now have the beginnings of an innovative new process and principles to back it up. We now coalesce everything we've written - the principles, the process. And then give it a distinct name so we can communicate the process.
We have a solid foundation from which to build our new process. We're at the end of the most significant innovation design phase in which we've used created from first principles. But it is just the beginning, and we'll need to iterate further to continually feedback and refine.
In the next and final part of this series, we'll look at the final process we've created that we'll called" Vodelling - A portmanteau of visual modelling. We'll dive deeper into that process, its supporting principles, and what it looks like.