Things I Learned the Hard Way Bryan Cantrill Oxide Computer Company

OXIDE The imperative to learn • The thrill of being a software engineer is that we are always learning • Learning is clearest with respect to our craft: learning new languages, new technologies – or new aspects of developing or deploying them • Learning extends beyond craft: software creation sits at the confluence of the technical and the creative, at once solitary and collaborative • The way we organize is not formulaic; we are always learning new ways of organizing ourselves and working with one another

OXIDE How we learn • Even though it can feel difficult at the time, learning is actually easiest when it’s highly structured – as in one’s formal education • But a formal education often suffers from being impractical – and much of the learning of an engineer will in fact happen on the job • Learning on the job will come from peers, from the organization, etc. • In this regard, it is an organization effectively teaching what it knows – and it’s the easy way to learn

OXIDE Things I learned the easy way • Coming up in OS kernel development at Sun Microsystems in the late 1990s and early 2000s, I learned a bunch of stuff the easy way: ○ The craft of C – and the importance of enforced standards ○ Source code control (and the bringover/modify/merge model) ○ Rigorous, first principles thinking – and the need to explain that thinking in the code itself (“Big Theory Statements”) ○ The primacy of postmortem debugging

OXIDE The hard way • But some wisdom comes from a gorier place: from doing it wrong • This is where an organization itself is often learning – and the imperative of the organization is not merely to teach but to learn • This is the hard way – and I have learned a lot the hard way! • Perhaps unsurprisingly: the human side of engineering has proven to have the hardest lessons

OXIDE The hard way: The primacy of testing • An operating system can be hard to test – and despite rigorous thinking, the difficulty of testing often prevents any unit testing whatsoever • In some early projects in my career we were far too late to develop a comprehensive test suite – and even then we made the mistake of effectively outsourcing the development of tests • The test suite became essential for us, but many tests needed to be rewritten (and some remain brittle to this day!) • Think about testing early and write your own tests

OXIDE The hard way: Invest in tooling • Tools are the things we build that we don’t ship – but that very much affect the artifact that we develop • It can be tempting to either shy away from developing tooling entirely or (in larger organizations) to dedicate an entire organization to it • In my experience, tooling should be built by those using it • This is especially true for tools that improve the artifact by improving understanding: the best time to develop a debugger is when debugging!

OXIDE The hard way: Debug by asking questions • We (unfortunately) do not really teach debugging methodology: more or less everyone learns debugging by doing it (often poorly) • It is tempting to believe that we engage in a kind of scientific method when we debug – but this is (in my experience) a myth • Instead of forming hypotheses, focus instead on informed questions: what do you want to ask of the system? • Summon creativity (and tooling!) to answer the questions – and then use the answers to questions to inform tighter questions

OXIDE The hard way: Understanding odd behavior • Computers are – at root – deterministic; software systems aren’t magic! • When something odd happens, it can be tempting to ignore it – especially true when it feels somewhat ancillary! • But odd behavior is worth understanding: at worst, it enhances our own understanding (that is, that the behavior is in fact expected)... • …but odd behavior can be an indicator of something much more deeply amiss – and in fact has represents an otherwise innocuous presentation of an important defect!

OXIDE The hard way: Half-measures on vexing problems • If a problem is reproducible, it can be debugged – however psychotic! • If a problem is not reproducible, it can also be debugged by asking questions of the state it left behind (core dumps, log files, etc.) • But what if a problem is both psychotic and non-reproducible? • On these (blessedly rare!) problems, take half-measures: change the system to leave more state behind, such that an additional occurrence of the problem will help bifurcate the search space for the root cause

OXIDE The hard way: Making technology choices • I have gotten in the most trouble when I am not deliberate about technology decisions – and choose something because it feels like a “safe” choice (“everyone uses it”) or because it wasn’t a choice at all • However, it can also be easy to overly deliberate on technology choices! • Technologies (and the decisions that they embody) reflect the values of their inventors, developers, users and proponents • When choosing technologies, weigh values heavily!

OXIDE The hard way: Predictions reflect the present • Many who make predictions – or are paid to make them! – do not revisit their past predictions; they are confidently wrong • The future is simply too dynamic to predict with repeatable accuracy • Many years of making, recording and revisiting predictions taught me that predictions tell you more about the present than the future • Be very careful about giving someone else’s predictions too much weight – especially if/when they feel wrong!

OXIDE The hard way: Bind a team with mutual trust That fall West had put a new term in his vocabulary. It was trust. “Trust is risk, and risk avoidance is the name of the game in business,” West said once, in praise of trust. He would bind his team with mutual trust, he had decided. When a person signed up to do a job for him, he would in turn trust that person to accomplish it; he wouldn’t break it down into little pieces and make the task small, easy and dull. – Soul of New Machine by Tracy Kidder (1981)

OXIDE The hard way: Fear causes introspection gaps • Fear is a base human emotion – and it causes us to take our most disruptive, most rash actions (fight-or-flight!) • Much conflict has fear at its root; when conflict erupts, it can be helpful to elucidate those fears • Must watch: Rachel Stephens’s excellent Monktoberfest 2022 talk!

OXIDE The hard way: Use values as a lens for hiring • It is essential for an organization to articulate its shared values • The most grievous mistakes I have made in my career are mishires • Interviews are deeply flawed as the sole mechanism for evaluating hires! • I have found that it is important that new hires help to reinforce and amplify shared values – and for that, values must be a lens in hiring • This is not easy! We have found a written process to be very helpful in this regard, and has helped inform our conversations

OXIDE Make the hard way the easy way? • We can make the hard way the easy way by articulating our experience • I have done this through talks over the years, but I have found that to be insufficiently dynamic… • We have found social audio (e.g., Discord) to be a great way to allow technologists to discuss their experiences – and convey their wisdom! • Check out the back catalog of our Oxide and Friends podcast… • …and if you start your own, let me know so I can like-and-subscribe!