I dislike [all] blanket statements, but complexity is never desirable.
Managing complexity is the most important technical topic in software development.
In my view, it’s so important that Software’s Primary Technical Imperative has to be managing complexity.
— Steve McConnell, 2004
It must have been around 2008 that I first read this quote in Steve McConnell's classic book, Code Complete, and it stuck with me all those years: "Software's Primary Technical Imperative" is managing complexity.
But what is complexity?
Complexity characterizes the behavior of a system or model whose components interact in multiple ways and follow local rules, leading to non-linearity, randomness, collective dynamics, hierarchy, and emergence.
— Wikipedia, 2024
How about a less complex definition?
Complexity is a property of an object with many parts and a complicated arrangement. It is also the state of being complex. Its opposite is simplicity.
— Wikipedia (Simple English), 2024
That's better.
So, if something has complexity it becomes more difficult to reason about because we must take into account various components and their interactions.
Different kinds of complexity
In software, we often split complexity into different forms: inherent complexity is complexity within the problem space, and incidental complexity is complexity in the solution space.
Let's illustrate the difference with an example.
You write stories and would like to publish them on the web.
At first glance, this has low inherent complexity. Let's go with that. The incidental complexity depends on how we choose to solve it.
⬇︎ Low. Sign up to any of the many online blogging platforms and start publishing your stories within minutes
◉ Medium. Write your stories in raw HTML and upload them to a web host.
⬆︎ High. Build your own SQL-backed CMS, host it somewhere, and publish your stories via the built-in editor
A problem with higher inherent complexity might have more requirements:
You write stories and would like to publish them to the web.
Stories are categorised by genre.
Stories are split up into chapters.
Some stories can only be read by subscribers.
Our first order of business is to constrain the requirements down to forms that can lead to simpler implementations. YAGNI is a good rule to follow, but may also be broken as with all rules. Things like budget, upcoming features, and other priorities need consideration.
In this example, it's not specified how stories are categorised. Clarify and aim to reduce inherent complexity in doing so — do we really need AI to do the categorisation, or can the user select a pre-defined genre when they submit the story?
For the sake of the example, let's finalise the spec:
Authors sign into the website.
When signed in, they can create a story and
add chapters to that story.
Authors can enter a text body into each chapter.
When they are finished writing, they can select a category
from a predefined list and publish the story.
They may also choose to limit the story to subscribers or
keep it public.
Users can view stories on the website.
They can see the full text of each story,
along with its chapter headings,
but cannot view stories that are for subscribers only.
A category can be selected to view stories relevant to the user.
Subscribers can sign in on the web page.
They can do everything that normal users can do,
but can also see subscriber-only stories.
The static HTML solution is definitely not powerful enough. There are probably online platforms with this feature-set, but they need to be evaluated for fit.
Solutions to this problem will, by definition, be more complex. But incidental complexity may still be considered anywhere from low to high depending on the implementation.
Let's say you've gone as far as to decide that you'll build a system yourself. Focusing on the author side of the equation, here are some ways incidental complexity might explode.
⬇︎ Low. You build on top of proven, mature technologies like Ruby on Rails and PostgreSQL. Authors can sign in, write a story, add chapters, select a category, select whether or not only subscribers can read the story, and publish it.
◉ Medium. You choose proven technologies and split the backend and frontend into separate apps: one is a Go API serving JSON from MongoDB, the other is a React frontend consuming that JSON.
Authors sign in and receive a JWT for future interaction; they can then start writing a story. Chapters are split up automatically based on the headings in the story body. The author can select one or more categories for each story, and may limit viewers to only users, subscribers, other authors, or a combination of the above.
⬆︎ High. You've identified several different problem domains and decide to implement each as a micro-service in Rust, which communicates via gRPC. You deploy each to its own AWS lambda, because we need to scale easily.
Stories are stored on a blockchain, because this proves who wrote them. Authors run native desktop apps, which they use to sign in via their cryptocurrency wallet.
Stories are generated via LLM, giving authors the option to modify the prompt and change the output. Categories and subscriber-only options are replaced with a proprietary algorithm that determines which content will be most interesting to each user.
It's worth noting that any aspects of each of these solutions can turn up the complexity; don't focus just on the systems architecture but also on the unnecessarily expanded feature-set as well.
So, that's the difference between inherent and incidental complexity: both can be rather easy to understand, but I believe they're more art than science; what one person may consider complex may be very different to another.
Abstraction
Let's look at abstractions. One way we typically manage complexity is abstracting it away. Recall earlier, when I said, of the simpler example:
At first glance, this has low inherent complexity. Let's go with that.
In fact, publishing stories to the web is an amazing feat of engineering — but only when you look below the surface. The web, strung together with DNS, HTTPS and web servers, is built on top of transfer protocols, the internet itself, and huge cables under the sea. We could keep going deeper.
So "publishing a story on the web" is actually enormously complex, when taken from this viewpoint. But if we had to think about all of that whenever we wanted to build a simple website, there would be no websites. Each of these things, that makes the web work the way it does, is an abstraction.
The web itself is an abstraction, which provides meaning and understanding on top of what is otherwise, to the human eye, incomprehensible. So too is the text you're reading now: it's just lines, dots, and squiggles; which are in fact abstractions on top of what is, at some level of abstraction, just binary.
And yet, though we humans share a lot of cultural context, we interpret some abstractions as different, or as more complex, than others might.
I find suckless software a fascinating concept. On the tin, it's "software that sucks less," driven by the idea that "all software sucks."
Have a read of the philosophy, and there are several references to complexity [of other software] and simplicity [of suckless software]. It opens with:
We are the home of quality software such as dwm, dmenu, st and plenty of other tools, with a focus on simplicity, clarity and frugality. Our philosophy is about keeping things simple, minimal and usable. We believe this should become the mainstream philosophy in the IT sector. Unfortunately, the tendency for complex, error-prone and slow software seems to be prevalent in the present-day software industry. We intend to prove the opposite with our software projects.
Suckless software, to me, is a subset of the Unix philosophy — do one thing well, small is beautiful, etc.
Suckless software often doesn't have options, or any configuration. If you want to change something, you have to edit the program and compile it yourself. If you want to add a new feature, or a "plugin" or some such, you have to patch the program and compile it yourself.
Now, I really don't want it to seem like I'm firing shots at Suckless. When I said I find it fascinating, I really didn't mean it pejoratively; it has its place, I've embraced it in the past, but it's not for everyone.
The point is, I think a lot of us would consider the need to recompile software to change some options as exceptionally complex. You need to make sure your architecture is supported, that you have the right (versions) of dependencies, know how to debug when you don't, generally know your way around a terminal, etc.
And yet by stripping the software down such that options and toggles and bells and whistles simply aren't a part of the codebase, it is arguably much simpler.
A similar example of different perspectives on the complexity of abstractions can be found in the static vs dynamic site debate, of which Jon Sully of Judoscale has already written an insightful post about.
Complexity in Life
So far we've talked about complexity in software, but the reason I really wanted to write about this was because I've lately come to understand that complexity is a great enemy in other areas of too, which we humans have been fighting for milennia.
I first landed on this idea a year or two ago, when reading Marie Kondo's The Life Changing Magic of Tidying Up. I know that not everyone's a fan, so please bear with me.
Marie Kondo writes that reducing clutter by selling, discarding, or better-organising your possessions leads to greater happiness. Somewhat hyperbolic and preachy, but I do think there's some truth to it.
I don't know about you, but my brain tends to feel overstimulated when I'm in a messy environment. Let's say I've been lazy and haven't yet done the dishes and they're strewn all over the kitchen. Every time I look in that direction, I subconsciously hook onto that mess and all sorts of thoughts enter my mind: "I need to tidy up," yes, but also "that mug was a really great gift," "it's a shame that that bowl has a chip in it," "I wonder what I'll have for dinner tomorrow," "I need to buy groceries," etc. And then, "I really need to focus on work," and stress.
It can be exhausting.
So: I tidy that stuff up. I get rid of the bowl with the chip, do the dishes, put stuff in a cupboard, create a reminder to do grocery shopping. My brain calms down and I can focus on my one important task: work. Complexity managed, simplicity restored.
Since reading Marie Kondo, I've been devouring self-help books. Sometimes they're great, sometimes they're awful, but more often than not I'll get at least one new idea from each book.
The one thread that I've noticed is almost universally common, albeit conveyed in different terms — from Marcus Aurelius' Meditations to Cal Newport's Digital Minimalism — is that simplicity — the absence of complexity — is important to one's wellbeing. Often it is written as Mindfulness.
I am not a mindfulness expert, but I broadly understand it as being aware of what is happening in the body and the mind presently, and working through that — simplifying — to your own benefit by casting off complex, intrusive, unnecessary thoughts.
Whether this is mindfulness or managing complexity or Buddhism or tidying up or some other thing isn't particularly important — I think the idea of urging towards simplicity in our ever-more-complex world has merit.
So, to round up: let's keep in mind that simplicity = good and complexity = bad as we try to leave the campfire of life a little bit better than it was before we found it.