{ "timestamp": "2013-08-06T12:30:00-0700", "title": "Distributed Systems Are Everywhere", "subtitle": "Where the full stack is headed", "location": ["Pivotal Labs", "San Francisco", "CA"], "author": { "name": "Josh Dzielak", "title": "VP Engineering", "company": "Keen IO", "email": "[email protected]", "twitter": "@dzello" } } 

About me * Full stack developer, going on 13 years * Currently VP Engineering at Keen IO * 4+ years of startups * 5+ years of enterprise consulting * Lead Engineer / Founding team Togetherville, 2009 (Disney) A Pivotal Client!! * Interested in distributed systems, high- performance, and linear scalability 

TL;DR The definition of a “full stack developer” is expanding to include a knowledge of distributed systems. 

given that – applications are processing more data... 

given that – the expectations of users and (product managers) are rising... 

given that – and Moore’s Law is in a funk, making it impossible to just ‘scale up’... 

Distributed systems are becoming the rule, not the exception. 

How do we define a distributed system? broadly speaking... In computing, a system distributed amongst “nodes”, usually independent computers interfacing over a network. 

As used today – A distributed system is a group of differentiated yet well-integrated components that solve a given problem at linear scale. Let’s look at each part. 

Linear Scale Processing throughput grows linearly with hardware. It’s one key characteristic that is bringing distributed systems into the spotlight. See: popular distributed databases 

Fault Tolerance Things break. Hardware fails. It’s inevitable. Distributed systems intentionally use nodes that share as little hardware/fault exposure as possible. 

Differentiation Distinct components with different jobs. Often exposed as “services” or APIs. (By this test, we wouldn’t leap to call a homogenous group of app servers a distributed system.) 

Integration Components within the system must be well-integrated (often loosely coupled). Usually this implies real-time and bi- directional interfaces. (By this test, we wouldn’t leap to call a batch-integrated group of legacy backends a distributed system.) 

Differentiation + Integration = Complexity! Complex systems are good. Complicated ones are not. 

I call these Complex-Linear Distributed Systems While there’s certainly more to them, linear scale is a common goal and a complex system is required to achieve it. 

RECAP * distributed systems are quickly becoming the norm * most distributed systems are complex, fault tolerant, and scale linearly * henceforth I’ll use “distributed system” to mean a “complex-linear distributed system” --- why is this important to me, a full stack developer? 

The next generation of full stack developers will speak distributed systems. 

What’s a full stack developer? There are many definitions. Here’s mine... 

A full stack developer can build an entire product (or company) single-handedly. 

ARE YOU WIZARD?? No, full stack developers aren’t superhuman. (but don’t tell them that) 

They’ve just been around the block a few times. In my experience, each lap of the block takes 2,000 to 3,000 hours. Developers who stick it out for 10-20k hours often reach the coveted ‘full-stack bad ass’ status. Full stack devs don’t always do *all* the work - they just could if they had to. 

Full stack developers know: user experience user experience user experience business value business value business value user interface user interface user interface (some) design (some) design (some) design MVC Request Request Action Action API’s data access data access business logic business logic services services data modeling data modeling data modeling data formats data formats data formats kernel network network I/O I/O shell 

Full stack developers can do: hiring fundraising mentoring pair programming user interface A/B testing user interface A/B testing test-driven API development test-driven API development database performance tuning database performance tuning linux kernel debugging linux kernel debugging 

That’s not enough. >:{ 

Every full-stack developer must also become a distributed systems developer. 

WHY? The definition of ‘full stack’ is the ability to build any level of the product or business. Businesses increasingly require distributed systems to be competitive. Full stack developers must become conversant and/or fluent in distributed systems. 

What does a DISTRIBUTED FULL STACK developer look like? 

No content

LOLJK Because of their experience, full stack developers can quickly acquire the tools and methodologies of distributed system building. 

More skills: monitoring stacks DEVOPS fault tolerance distributed coordination partitioning serialization more about networking I/O and performance shell scripting 

Distributed FS developers will need to learn: a) common software design heuristics for distributed systems: partitioning, serialization, fault tolerance, tradeoffs (like CAP), performance, capacity... path-oriented design: model the system as interdependent paths (e.g. read path / write path) and distribute complexity via priority 

Often, performance is the essential ‘feature’ of the system/product. And thus, it becomes the key driver of the design. This is very different than traditional web/ app work where performance optimization can come later. Insufficient performance can undermine everything in the 11th hour #realtalk 

Latency Numbers every distributed full stack developer should know execute CPU instruction 1 ns fetch from memory 100 ns send 2k over a 1Gbps network 20,000 ns read 1MB from memory 250,000 ns spinning disk seek 8,000,000 ns read 1MB from disk seq. 20,000,000 ns send packet US->UK & back 150,000,000 ns SSD disk seek 150,000 ns read 1MB seq. from SSD 1,000,000 ns round trip in same DC 500,000 ns run unit tests 60,000,000,000 ns github page load 161,000,000 ns hire a full stack engineer 2E+15 ns raise a round of funding 7.7E+15 ns From Norvig, others These numbers are guidelines and YMMV. You’ve been warned! 

Distributed FS developers will also need to learn: b) a lot of “devops” provisioning; be it chef, puppet, or salt. oh, and vagrant (with vmware) monitoring; new relic is great, but it won’t be enough. embrace it. make friends with devops and ops folks. you will need their tools to have any idea of how your complex system is behaving. 

You want one of these torx is a water-cooled hackintosh w/ 32GB RAM, SSD RAID, and 4960x1600 resolution 

RECAP full stack developers have the smarts and panache to become DISTRIBUTED complex-linear full stack developers * learn new design heuristics * embrace devops 

RECAP by embracing distributed system development, you can prevent many problems of scale AND open up new opportunities for feature and product development that wouldn’t be possible otherwise (lots of these have to do with data) 

Practical ways to get down with distributed 1) Try out a distributed database. 2) Do a project with Storm or a similarly well-abstracted distributed system. 3) Integrate automated provisioning into your development workflow. 4) Chaos monkey ideally fault-tolerant parts of your existing system. 

Resources Latency number gist https://gist.github.com/jboner/2841832 Teach yourself to program in 10 years http://norvig.com/21-days.html What is a full stack developer? http://www.laurencegellert.com/2012/08/what-is-a-full- stack-developer/ The Full Stack, Part 1 https://www.facebook.com/note.php? note_id=461505383919 

Thanks! Questions? Talk at my face or email me at [email protected]