Oral tradition in software engineering: Passing the craft across generations CTO [email protected] Bryan Cantrill @bcantrill

Oral tradition • Long before humans developed systems of writing, we were leaving legacies of information via oral tradition • Oral tradition has been used to: • Preserve and explain cultural values • Convey skills and wisdom • Entertain as well as enlighten • Oral tradition continues to play an important role in post- literate societies — including in software engineering

Oral tradition in software engineering • In the beginning, software was subservient to hardware • Software rarely enjoyed the levels of design documentation found in computer design • Like true oral tradition, it relied primarily upon oral retransmission — and much of this is lost today • Fred Brooks’ Mythical Man Month captures the tribal wisdom of this antediluvian era of software

Oral literature • When oral tradition is distilled into clear narrative and reliably repeated, it becomes oral literature • For us in software engineering, the early Internet allowed a way to reliably share stories via USENET • USENET is filled with incredible and interesting narrative, but most survives only in archive • But one story from USENET does survive — and it has become the epic of software engineering…

Our epic, as penned by Ed Nather A recent article devoted to the macho side of programming made the bald and unvarnished statement: Real Programmers write in FORTRAN. Maybe they do now, in this decadent era of Lite beer, hand calculators, and “user-friendly” software but back in the Good Old Days, when the term “software” sounded funny and Real Computers were made out of drums and vacuum tubes, Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Machine Code. Raw, unadorned, inscrutable hexadecimal numbers. Directly.

…and our epic hero, Mel Kaye Lest a whole new generation of programmers grow up in ignorance of this glorious past, I feel duty-bound to describe, as best I can through the generation gap, how a Real Programmer wrote code. I'll call him Mel, because that was his name.

Our Odyssey • The Story of Mel was recorded in the early 1980s, but dates from the earliest days of software in the 1960s • Another set of stories recorded in the late 1970s at Data General has become software engineering lore… • Tracy Kidder’s Soul of a New Machine has wisdom that has long survived the team and company it describes • If Mel is our Illiad, Soul is our Odyssey

The 1980s and 1990s • In the deeply proprietary era of the 1980s and 1990s, oral tradition was locked on proprietary islands — tribes privately passed on their own traditions and practices • Collections of oral tradition like sgi.badattitude were the stuff of legend — but they too have been lost • The canonical folklore from this era may be John Browne’s beautiful The Bug Count Also Rises, the winner of an internal Microsoft Hemingway parody contest (!)

Dawn of a new era: November 7, 2005

Oral tradition post-YouTube • With the rise of Internet video, story — orally delivered and aurally received — could travel unencumbered • Stories could be suddenly retold simply by sharing a link — with all of the force and authority of the original • Oral tradition can spread so quickly, we explain it in terms of a pandemic: stories that become viral • We in software were particularly lucky in that Google Tech Talks managed to capture much oral history and tradition before participants recognized what it meant to be recorded

For me personally, a watershed

For me personally, a watershed (ahem!) DTrace Review

Oral tradition post-YouTube • Post-YouTube, our oral tradition is suffering from overload — we have an embarrassment of riches! • The true oral tradition has shifted from the content to the meta-content, e.g. this evergreen topic on HN:

Oral tradition closer to home • Every source base is awash in narrative: of origin, of struggle, of conflict, of rewrite, of success and of failure • These narratives ultimately convey not only the history but also the values of a source base • In the proprietary era, these were purely oral tradition — something to regale new hires with over lunch • Every source base is loaded with these; several that are relevant to mine… (UNIX Unix 6th Edition 7th Edition SunOS System III SVR4 Solaris OpenSolaris illumos/SmartOS!)

I: The untimely death of Joe Ossanna • In 1973, Joe Ossanna at Bell Labs wrote troff in PDP-11 assembler for the Graphic Systems CAT typesetter • In 1975, Ossanna ported it to C — with the intent of a later clean rewrite to make the macro language less alien • In 1977, Ossanna died suddenly, of either a heart attack, a car accident or (most graphically) being struck by a car • The code he left behind — which remains in our source base — is inscrutable; Brian Kernighan failed to rewrite it! • Code survives us

II: “You are not expected to understand this” • Perhaps the most (in)famous comment in Sixth Edition:
 
 
 /*
 * If the new process paused because it was
 * swapped out, set the stack level to the last call
 * to savu(u_ssav). This means that the return
 * which is executed immediately after the call to aretu
 * actually returns from the last routine which did
 * the savu.
 *
 * You are not expected to understand this.
 */
 if(rp->p_flag&SSWAP) {
 rp->p_flag =& ~SSWAP;
 aretu(u.u_ssav);
 } • That which we can’t explain, we don’t understand!

III: Logical XOR • I have always been enraged disappointed by C’s lack of logical XOR, having had a hard time holding my tongue, e.g.:
 
 
 /*
 * Logical XOR, where are you?
 */
 ASSERT((vstate->dtvs_nglobals == 0) ^ (vstate->dtvs_globals != NULL)); • I became involved in a heated argument with Roger Faulkner, who insisted that Dennis didn’t add logical XOR because “it can’t short-circuit” — which is obviously stupid • Important difference between Roger and me: Roger had worked at Bell Labs with Dennis…

III: “Isn’t that much worse to write” (!) From [email protected] Sun Aug 6 23:33:54 2000
 To: [email protected]
 Subject: Re: history question
 
 Hi Roger,
 
 back from some vacation; here's a belated response.
 ^^ was omitted because it is of little use, particularly
 because the "shortcut evaluation" guaranteed for || and &&
 doesn't help--both sides have to be evaluated no
 matter what. When you do need the effect,
 (A!=0) ^ (B!=0) isn't that much worse to write.
 
 Dennis

IV: “You think this is cheesy?” • A favorite comment in the operating system is one that Jeff Bonwick and I wrote together in a moment of crisis • We had (finally!) figured a narrow path by which we could solve a devastating problem that seemed insurmountable • The details are highly technical, but documented thoroughly in code comments — the context around it in an abstruse chapter I wrote in Beautiful Code…

IV: “You think this is cheesy?” /* * The turnstile hash table is partitioned into two halves: the lower half * is used for upimutextab[] locks, the upper half for everything else. * The reason for the distinction is that SOBJ_USER_PI locks present a * unique problem: the upimutextab[] lock passed to turnstile_block() * cannot be dropped until the calling thread has blocked on its * SOBJ_USER_PI lock and willed its priority down the blocking chain. * At that point, the caller's t_lockp will be one of the turnstile locks. * If mutex_exit() discovers that the upimutextab[] lock has waiters, it * must wake them, which forces a lock ordering on us: the turnstile lock * for the upimutextab[] lock will be acquired in mutex_vector_exit(), * which will eventually call into turnstile_pi_waive(), which will then * acquire the caller's thread lock, which in this case is the turnstile * lock for the SOBJ_USER_PI lock. In general, when two turnstile locks * must be held at the same time, the lock order must be the address order. * Therefore, to prevent deadlock in turnstile_pi_waive(), we must ensure * that upimutextab[] locks *always* hash to lower addresses than any * other locks. You think this is cheesy? Let's see you do better. */

Code comments as oral tradition? • The best comments seem to have an aspect of oral tradition to them: they not only serve to explain what is going on, but provide a narrative as to why • Comments live with the code but do not otherwise affect its correctness — allowing for broad scope and artistic license • Relevant, correct software systems will survive long past their creators; comments are the art within the timeless machine • Comments are how we speak to future generations!

Oral tradition in software engineering • Software engineering has changed radically in the last decade: it’s now open, transparent and distributed • This has forced a revolution in oral tradition, which has increasingly become recorded, yielding oral literature • Oral literature allows us to expose and preserve the narrative behind the artifact — to explain the why of the thing • Software engineering teams should be encouraged to record their oral tradition: it attracts technologists who share values! • Our oral tradition is part of our legacy to future generations!