Where are the software cartographers? (Part 1)
Nearly every complex man-made system has accurate, up-to-date, useful maps that reduce complexity... That is every complex man-made system, except for software systems. You can navigate a city with a map, build a building with architectural drawings, and construct complex machines with technical diagrams. However software has no standardised process for visualising a code base. This lack of visualisation creates complexity that leads to a cycle of software rewrites and engineers becoming overwhelmed. But the solution is simple: we need maps of our working software and someone to keep them in sync with our code.
In this post, I will show the problems caused by a lack of maps, how maps solve them and introduce a change you can make today to improve your teams' mapping.
A thought experiment on complexity
Let's perform a thought experiment:
I ask you to create me a map of a small village. In the morning, I take you to the village. I've given you a pen and some paper to draw the map. On the map, you must draw the approximate layout of the village (roads, street names, parks, etc.) and the general use of each building (shop, house, restaurant, etc.).
How long would it take you to map out the village? You'd probably be finished before lunch. Given some more time, you'd even be able to memorise the whole village.
It would present a much bigger challenge if I asked you to create a map of your country's largest city. It would likely take you months or years to map out the city. However, you would never be able to memorise the whole city.
How code bases are like villages, towns and cities
Villages, towns and cities are a lot like software. A small code base is like a quaint village: Simple and easy to understand completely. A large code base is like a busy city: Complex, hard to understand, and unknowable for any single person.
Why can we understand villages yet fail when it comes to cities? Well, because of our cognitive limitations
- A village is within our capacity. It has fewer complexities, so we can easily digest it.
- A city exceeds our capacity. It has a much more significant amount of data than we can digest in any reasonable amount of time.
How do we handle the cognitive complexity of a city in real life? We use maps (and diagrams, signposts, etc.). Maps simplify the city so we can navigate and understand it without having to pay attention to irrelevant detail. Without maps, we wouldn't be able to make sense of a city because there's too much to know. It’s the same for code: the more complex a code base becomes, the more you need a map to abstract that complexity.
A map is an abstraction. It abstracts a system in visual form. Maps simplify reality, to make it manageable for our minds. You will already be familiar with some maps we use in software engineering: Architecture, domain and flow diagrams, white-board drawings, etc. You will also be familiar with the fact that maps are seldom kept up to date with the live system and are mostly useless in day-to-day engineering.
Maps help you understand the territory
Code is the territory. To understand the territory, you need to be an expert programmer. But even as an expert, code takes time to understand. That time must be invested in building up a mental map of how the code works.
Whilst the territory - the code - is complex and hard to understand immediately, maps are easy to understand: A map is a simple abstraction that hides complexity and unnecessary detail. They’re the GPS for your development process.
Maps are a better way to learn a code base than relying solely on your mental model. A mental model is abstract (it cannot be seen), varies widely from person to person, and takes a long time to build up into a cohesive picture. In contrast, a map is concrete and visible, aligns many minds with shared understanding, and is quickly understood.
Cognitive limitations and maps
Given a long-enough period of time, every project will exceed your ability to understand it. If you don't simplify your project, you're going to forget, you're going to get overwhelmed, you're going to slow down, and you're going to get frustrated.
One of the traps we fall into that prevents the creation of maps in the first place is, near the beginning, we don’t see the upcoming complexity. We can’t grasp the moment complexity will exceed our ability. So, the point when a map is at its most obvious usefulness is also the point when it would be tough to create.
Creating maps from the start would prevent us from falling into the trap of being overwhelmed by complexity. If we avoid creating maps, this complexity permeates throughout the code base until we reach the ultimate stage of many projects: the rewrite.
The rewrite lifecycle of a project
Rewriting software at the project level frequently happens due to not having maps. It happens when the team are overwhelmed, so they start some or all of their project again. This cycle is experienced continually throughout the life of a code base.
Let's get to our example: We'll continue with the analogy of code bases being like villages, towns, and cities. The cycle goes like this:
The Village
The ‘village’ is a small code base - every engineer could understand the whole code base.
A project often starts with a map of some kind: Diagrams to show the flow of the system, domain diagrams which capture the relationships, architectural diagrams to show how the system fits together with others, and more. The start is also when the engineering, product, and UX/Design teams work closely together to create the first iteration of the application.
As soon as the engineering team writes the first line of code, the code becomes the source of truth, and the village is created. From this point onwards, only professional programmers can understand the system. Even though non-programmers usually drive the system’s requirements.
Within weeks, diagrams that laid the foundation for the project begin to rot. Unbeknownst to the team, this deterioration lays the foundation for the application's demise.
The Town
The ‘town’ is a medium-sized code base - no individual understands the whole code base, but the team as a whole could fully understand it.
The application has grown past the village. The team may be split into distinct groups to handle the increased complexity. Efficiency reduces as fewer people understand different parts of the code base, and it takes time to upskill the engineers on new parts of the system.
Whilst the town is more complicated than the village, parts of the application can still be rewritten, reworked and evolved separately, as the team can pool their thinking to solve the increasingly complex problems.
However, without a map, things begin to slow down. Non-engineers are blind to the workings of the system. This lack of visibility increases the need to communicate with the engineers for clarification. The engineers struggle to articulate exactly what the system is doing because it takes time to analyse the code. This back-and-forth acts as an additional drag on the team, often in the form of meetings.
The City
The ‘city’ is a large code base. The team, collectively, could not understand the whole code base - it exceeds everyone's abilities
The code base is now over-complicated. The over-complication often happens abruptly when someone leaves the team, taking with them valuable knowledge. Now, the codebase has exceeded the combined cognitive ability of the team. The code base has become fragile. Engineers are less motivated as the difficulty has increased. The engineers don’t understand the existing system, so evolution must happen on top of existing code. The reduced ability to change means mistakes become irreversible as parts of the system become rigid. New engineers complain about the difficulty. The engineering team even avoids part of the code that they no longer comprehend.
There’s no map to guide them through this phase, no abstraction that would simplify the system for them. Thus everyone on the team begins to accept there’s only one course of action: The rewrite - either a complete or partial one.
The Village (again)
The team have arrived back at the start - "This time, we'll get it right" - they proclaim. And the cycle begins anew - A little wiser and better informed but ready to repeat the cycle. With no maps, the same outcome will almost be assured.
Stopping the rewrite cycle
Complexity is unavoidable as a system develops. We shouldn't try to prevent this inevitability; we should instead handle it by making it understandable. Thus, there is nothing wrong with cities (large code bases). A city represents the inevitable complexity of any system. The complexity would be similar even if it were a complex system of 100s of villages (microservices). Instead, the complexity would shift to the roads (communication layers) between the villages (other microservices).
A map prevents your team from being overwhelmed by complexity. If the cycle above had a map, the issues raised (e.g., cannot rework code, lack of understanding, increasing communication, etc.) would be diminished. The map allows continual navigation of an increasingly complex system in perpetuity as long as the map is in sync with the code base.
The biggest issue with maps is keeping them in sync with the code. When documenting code, a similar problem exists; Code often goes out of sync with the code base. There is even a principle I created to help with that "documentation should be close to the code". But keeping a map close to the code is ineffective because our tools aren't good enough. To keep them in sync, we'll need to change people's behaviour by understanding the current problem and then create a new process with the help of principles.
In large codebases without maps, there are three root cause problems that every team experiences:
-
Understanding decreases as complexity increases. People reach their cognitive limits, first as individuals and then as a team.
-
The number of potential problem solvers diminishes as complexity increases, so the team has access to less cognitive power to solve problems as the system is less visible to everyone.
-
The ability to change the system reduces; it becomes harder to evolve as it takes on a more rigid form. This rigidity prevents adaptation to future change. Big mistakes may already be baked in.
There's also one last general problem that is endemic in Software Engineering. It is why this cycle keeps happening again and again, whereby we accept unmanageable levels of complexity in our system because: we don't see our limitations.
Why don't we see our limitations? Stories of the fabled 10x engineer (which I think is a highly negative and unrealistic view of people), the culture of "being smart", and a tinge of "can't be wrong", and —well— we end up where we are now.
Embracing our limitations makes us better. Embracing our limitations, far from weakening us, creates innovation by recognising the need to solve human problems rather than just technical ones. These human problems, when solved, make outsized returns to how people engineer. If we keep solving technical ones, such as adopting a new programming language or complicated framework, it does nothing to prevent us from repeatedly exceeding our limitations.
In this case, accepting limitations allows us to look for new ways to reach our goals, such as simplifying the systems by creating maps. If people cannot understand a system, what hope do we have of making a system better in the long term?
Towards the creation of a Software Cartography process
Software engineering needs a new way of thinking: that of a Software Cartographer. Teams should be willing to describe the system at a higher level of abstraction and keep that map in sync with the code. If we can achieve that, we can create a more evolvable system that is better understood, doesn't need to be rewritten, and prevents engineering teams from being overwhelmed.
We all have many of these mapping methods at our disposal already:
- Architecture diagrams
- Diagrams to show the flow of the system
- Domain diagrams which capture the relationships.
Keeping your maps up to date is essential to making better software. Move your diagrams onto low-friction tools such as Miro or Lucidchart, which support multiple collaborative users. Shared diagrams allow numerous users to work together, and the reduced friction increases usage. The next step is to construct a process that keeps everything in sync allowing your engineers to spend time working at higher levels of abstraction (further away from the territory/code). The process will need incentives to work, which is down to your team or organisation to figure out what works for you.
In the next part of the series we'll look at how first principles works, and then we'll take a look at the low-level principles and behavioural changes you'll need to create a mapping process. To get inside the Software Cartographer's mindset, we'll approach this mapping process from the bottom up. The resulting process I call Vodelling - Its purpose is to increase understanding, aid rapid evolvability, and allow more minds to work together. It's simple, powerful and human-focused. Part 2
If this problem deeply speaks to you and you want to help solve it, drop me an email https://adamcraven.com/contact/ or reach me at twitter