creation of abstraction • Abstractions allow us to build software systems that do sophisticated things; they are the shoulders we stand upon – and provide to others • Abstractions are qualitative; good ones allow us to hide gory implementation details – to build castles atop them and tunnels beneath • Bad ones, however, leak: instead of sealing implementation details, they seep them – yielding systems that are unwieldy and brittle
do not exist at all – we have complexity • Fred Brooks famously called this accidental complexity, differentiating it from the essential complexity endemic to a particular problem • Complexity doesn’t merely accrue, however – it can explode… • Uncontained accidental complexity in one component can become essential complexity in something that must interact with it!
tamed from within existing systems, this is decidedly uphill; most changes to complexity happen in new systems • Some systems are broadly emergent: they are (implicitly or explicitly) a reaction to accidental complexity in systems that came before them • Other systems are more deliberately engineered, attempting to take on essential complexity in the problem itself from first principles • The abstractions offered by the system may be orthogonal to its implementation, and themselves may range from simple to complex
essential complexity and complicated abstractions – often to serve larger functionality • Big with respect to scope, people, and time – it’s software-in-the-large • Complexity is a killer for these systems, often literally: they are so large that they are afflicted by (e.g.) scope creep, second-system syndrome • Our most essential and most foundational software (operating systems, compilers, databases) necessitate constructed systems…
all constructed systems – and the complexity that afflicted these systems was top of mind • Much of the thinking on software engineering through the 1990s is on effectively managing these risks • The words of Brian Kernighan are representative: “Controlling complexity is the essence of computer programming.” • Writ broadly, without the lodestar of simplicity in abstraction, constructed systems struggle to know how to say what they won’t do
constructed software systems often leave technologists asking: “does it have to be this complicated?!” • This leads to rebellious systems, ones that overthrow what came before them, often by discarding unnecessary constraints • These systems think of and present themselves as who they are not almost more than who they are, inciting passion and inspiring followers • Examples: Unix (v. Multics), C (v. PL/I), RISC (v. CISC), Rails (v. J2EE), PDF (v. PostScript), microservices (v. monoliths), Node (v. Python/Ruby), NoSQL (v. SQL), Talos (v. Ubuntu/Rocky)
as rebellions that it becomes tempting to think that any rebellion will succeed… • Rebellious systems fail when – in their attempt to eschew complexity – they discard not just accidental complexity but also essential complexity • This can result in more complexity, not less – and worse, complexity no one talks about for fear of perceived loyalty to legacy systems • This is microservices turning into “distributed monoliths” – or the failure of unikernels to meaningfully cross into production systems
one would design from first principles – they are by their nature compromises with time or space • Often, accreted systems were born elsewhere: they are constructed systems (or even prototypes!) that congealed – or rebellions that fizzled • Some software was born accreted, e.g., ACPI, UEFI, BMC, IPMI • The challenge with these systems is that they become constraints on the systems around them, ossifying us in a web of complexity
born elsewhere, we know their accumulation into accreted systems as something else: technical debt • For systems that are born accreted, however, the failures are predictable: bug-infested death marches to nowhere • On the one hand, one could argue that the presence of accreted systems is a kind of failure – but this is too reductive… • Accreted systems – at some level – work well enough; their problem is more the sedimentation of complexity that they represent
complexity challenges of constructed systems • We have endured rebellious systems becoming accreted ones – often with the concomitant complexity that they were putatively overthrowing • And so much of the work that we do is with or on accreted systems! • All of this is enough to induce complexity despair – to view complexity as a kind of entropy that only increases, propelling us all to heat death
innovate with respect to the abstraction itself – the problem that they are solving includes the abstraction • These are revolutionary systems, which are often (but not always!) developed by small, focused teams – over an extended period of time • Example systems: CDC 6600, System R, TCP/IP, Macintosh, AS/400, PostScript, NeXTSTEP, Python, NFS, TeamWare/BitKeeper, Plan 9, HTTP, Java, Disco/VMware, SQLite, LLVM, x86-64, iPhone, ZFS, DTrace, S3, EC2, Go, TypeScript, Docker, Rust, Oxide
long time to develop (nearly always longer than its creators believe!) – and it can be unclear when to ship • Worse, adoption is slowed by the new abstraction, as it necessitates upgrading the trickiest software to upgrade: the software of the mind! • And plenty of would-be revolutionary systems miss the mark, providing an abstraction no one wants – or solving a problem that no one has • The biggest challenge for a revolutionary system is to stay funded long enough to effect the revolution it envisions for itself
because they tackle essential complexity in a way that allows others to abstract it away • It follows that developing revolutionary systems often requires taking on extraordinary amounts of essential complexity • The roots of revolutionary systems can often be found in experience with accreted systems – may such experience stoke the revolutionary in each of us!
we’re drowning in complexity, remember that complexity is not entropy – revolutionary systems do exist! • Revolutionary systems come from working on accreted systems, which become a gory education in accidental and essential complexity • Similarly, rebellious systems often come from working on constructed systems – and discovering essential complexity that is in fact phantom • It is very complicated to make things simple!
2011 talk, Simple Made Easy • Fred Brooks’s timeless 1986 essay, No Silver Bullet – Essence and Accident in Software Engineering • Dave Pacheco’s 2025 OxCon talk, Update on Update • Oxide and Friends, No Silver Bullet and Systems Software in the Large • Future of Coding episode 62, No Silver Bullet
bcantrill
iamctodd
eileencodes
sferik
andyhume
csswizardry
axbom
tamaranovitovic
techseoconnect
baasie
kneath
tammyeverts
