Zebras all the way down The engineering challenges of the data path CTO [email protected] Bryan Cantrill @bcantrill

The luxury of statelessness • In service-oriented software systems, we love statelessness • And for good reason: stateless components — like finite state machines — lend systems many desirable properties! • Stateless components can be easily made immutable, scalable, re-deployable, restartable, upgradeable, etc. etc. • Of course, persistent state still very much exists — we just use separation of concerns to confine the management of state to those services that do it explicitly and exclusively…

The data path • The data path consists of the software, hardware, and firmware components between a service endpoint that offers persistence and the implementation of that persistence • The data path always ends in non-volatile storage, which (for now, anyway) means either flash or magnetic media • The data path traverses many subsystems and components — and nearly always is a distributed system itself • We place great demands upon the data path…

The demands of the data path • A data path that merely works much of the time is insufficient • We (rightfully) expect perfection from the data path: we expect it to be consistent, available and partition-tolerant! • Of course, Brewer’s CAP theorem tells us that this isn’t actually possible — we must make tradeoffs • Even a well-engineered system can’t beat CAP — but a poorly engineered one will be flailed by it, becoming pathologically unavailable or inconsistent • Zebras are the difference

Zebras? • In American medical slang, a zebra is a rare and exotic condition that can be conflated with more common ailments • Medical students and residents are cautioned against diagnosing them, to the point of aphorism: “when you hear hoofbeats, think of horses not zebras” • But — as anyone who has been afflicted by one will affirm — zebras emphatically exist!

A zebra close to home

Zebras in the data path? • Even though the data path runs on and ends with hardware, it consists of many disjoint and unseen software components • The paradox of software (especially that of the data path!) is that software is both information and machine • When software works correctly, it survives as information does: namely, in perpetuity • Especially where software is expensive to write and difficult to fix, there is an overwhelming bias towards extant software • Over time, the horses are found; only the zebras are left

Hunting zebra • We must assume that unusual pathologies — especially in a distributed system — will not be readily reproducible! • When we are culturally afflicted with “bias for action”, it becomes tempting to immediately change the system to fix it • This is the wrong first motion: the choice to restore service versus understanding it is often a false dichotomy! • We must not change the system but rather observe it — we must focus not on snap hypotheses, but rather initial questions • The observability of the system is paramount!

Observability at Joyent • Observability is an organizing principle at Joyent — it is a primary reason that we run SmartOS, our illumos derivative • Manta — our (open source, container-centric) object storage service — has SmartOS and ZFS at its core • Manta uses sharded PostgreSQL for metadata (+ ZooKeeper for leader election), with services primarily in node.js • We invested heavily in the observability and debuggability of node.js — and it is a (the?) reason we still use node.js

Observability at Joyent Samsung! • Out of desire to build their own cloud based on Manta and Triton (our open source cloud management system), Samsung bought Joyent in June 2016 • While Manta has been in production for several years, Samsung’s level of scale has brought new-found challenges • Good news: between several years of production + observability (logging, DTrace, mdb) + hyperscale post-Samsung, we have nailed many thorny problems in Manta • Bad news: our stack — and that of every data path — has components that we still struggle to observe and debug…

Zebra sanctuary • Unfortunately, the data path is laced with proprietary software that can’t be observed, audited, verified, or debugged • This is the software that interacts so directly with the hardware as to create the illusion of hardware to higher-level software • This is firmware, and it runs so dark and deep in the data path that much of it is impossible to see or catalogue • Firmware that operates silently will also fail implicitly — it is hardware failing with software’s failure modes

Zebras in the spindle • Rotating magnetic media is a modern mechanical marvel • With sealed enclosures and helium-based drives, densities continue to increase — the disk will be with us for a long time! • Disks are vulnerable to vibe, temperature, particulates, aspersions, wear, etc. — magnetic media will fail! • But the disk knows this, and sophisticated on-head/on-controller firmware steers around failed media… • …leaving much nastier failure modes

Zebras in the spindle • Disks can (emphatically!) read or write the wrong data • Seeing this coming reality in the early 2000s, ZFS was designed around total data path integrity via indirect checksums • ZFS has discovered all manner of data corruption in storage systems putatively too expensive to suffer such problems… • And yet even ZFS oversimplified the failure modes of disks: 15+ years of deploying ZFS, we have seen disks fail in much more exotic ways than we thought possible

Zebras in the SSD • Flash wears out so frequently and quickly that much of an SSD is managing wear and mapping operations to functional flash • There are entire universes of system software in every SSD! • SSDs have incredible variety in their operating envelopes — and can accordingly fail in wildly divergent ways • This can represent systemic risk in that many SSDs can fail in the same way at the same time… • Confession: We’ve been so concerned about a flashtastrophy that we have always grossly over-engineered our own SSDs

Zebras in the HBA • The host bus adapter is responsible for brokering I/O from the operating system to the physical devices • This is more complicated than it might seem — and in particular, HBA firmware is infamous for losing I/O under load • From the perspective of system software this will be an I/O that never returns — which means it will be timed out and retried • While the system will maintain liveness, this will induce a latency outlier — which can manifest itself far up the stack (e.g., TCP resets!)

Zebras in the DIMM • DRAM is a capacitor that must be periodically refreshed • DRAM is susceptible to fatal failures (e.g., corrosion due to humidity, temperature or other environmental failures) • As the speed and density of DRAM have increased (and the voltage has dropped), DRAM has become more susceptible to transient bit failure not due to any hardware malfunction • The “Firmware First” (!) model of error handling in x86 (and the demise of CMCI) is leading to a silent epidemic of DIMM failure!

Zebras in the chassis • Even the chassis itself is not immune from software failure • For example, software and firmware control fan speed — and failures in that software can result in fans stuck running at their highest speed • Fans are not designed to run at full power for extended periods of time; they wear out or (worse) induce vibration in the chassis • The effects of (say) vibration will be felt far from the source — and again, may only manifest latency not explicit failure

Zebras in the NIC • Failure in the network interface card can be due to NIC firmware failure or hardware failure (e.g., the optical transceiver) • Networking failure should be entirely survivable by a distributed system, but that doesn’t mean it’s without consequence! • Use of the link aggregation control protocol (LACP) seems tempting — but can requires more sophisticated software in the switch (i.e., MLAG)… • …which itself can lead to new failure modes!

Zebras in the top-of-rack switch • As their own complicated ecosystem of software and firmware, top-of-rack switches are prone to software failure • Failure in the top-of-rack (or worse, the L3 core) can have an enormous blast radius in a distributed system… • For example, a switch that drops its ARP tables can result in a distributed system going massively split brain… • Or a switch that gets stuck broadcasting traffic can easily DDOS an entire distributed system — revealing that there is a single- point-of-failure after all!

Zebras all the way up • These problems do not manifest themselves cleanly at the point of origin for reasons both pragmatic and economic • Hardware vendors don’t want gear shipped back for RCCA! • Arguably, unreliable components allow (force?) upstack software to discover its novel failure modes • But that is an argument for debugging and resolving those (additional) problems upstack, not for unreliable components!

Don’t fear the zebra • The data path is not to be undertaken lightly • Do not assume that testing and monitoring can substitute for system understanding; enshrine observability • Reward complete understanding, not merely resolution! • As long as it’s unobservable, firmware is the enemy — and trends toward sophisticated firmware are especially troubling! • Open source software affords us a quality ratchet: we shouldn’t spend our careers re-solving the same problems!

Further reading and viewing • For an enlightening (and more positive) take on firmware, check out the amazing videos of Micah Elizabeth Scott (@scanlime) • For a snapshot of what we’re currently working on and thinking about with respect to Manta/Triton, see the Joyent Requests for Discussion (RFDs) — especially RFD 89 (“Project Tiresias”) • For more on node.js debuggability, see Dave Pacheco’s talk on “Industrial-grade node.js” • Also, thank you to Amanda Lundberg of White Coat Captioning for the superhuman real-time captioning!