Distributed Systems Are Everywhere - Where the full stack is headed

Distributed Systems Are Everywhere - Where the full stack is headed

The definition of a “full stack developer” is expanding to include a knowledge of distributed systems.

Aee8ace6215b362ce4524bfdfc4a718c?s=128

Josh Dzielak

August 06, 2013
Tweet

Transcript

  1. 1.

    { "timestamp": "2013-08-06T12:30:00-0700", "title": "Distributed Systems Are Everywhere", "subtitle": "Where

    the full stack is headed", "location": ["Pivotal Labs", "San Francisco", "CA"], "author": { "name": "Josh Dzielak", "title": "VP Engineering", "company": "Keen IO", "email": "josh@keen.io", "twitter": "@dzello" } }
  2. 2.

    About me * Full stack developer, going on 13 years

    * Currently VP Engineering at Keen IO * 4+ years of startups * 5+ years of enterprise consulting * Lead Engineer / Founding team Togetherville, 2009 (Disney) A Pivotal Client!! * Interested in distributed systems, high- performance, and linear scalability
  3. 3.

    TL;DR The definition of a “full stack developer” is expanding

    to include a knowledge of distributed systems.
  4. 6.

    given that – and Moore’s Law is in a funk,

    making it impossible to just ‘scale up’...
  5. 8.

    How do we define a distributed system? broadly speaking... In

    computing, a system distributed amongst “nodes”, usually independent computers interfacing over a network.
  6. 9.

    As used today – A distributed system is a group

    of differentiated yet well-integrated components that solve a given problem at linear scale. Let’s look at each part.
  7. 10.

    Linear Scale Processing throughput grows linearly with hardware. It’s one

    key characteristic that is bringing distributed systems into the spotlight. See: popular distributed databases
  8. 11.

    Fault Tolerance Things break. Hardware fails. It’s inevitable. Distributed systems

    intentionally use nodes that share as little hardware/fault exposure as possible.
  9. 12.

    Differentiation Distinct components with different jobs. Often exposed as “services”

    or APIs. (By this test, we wouldn’t leap to call a homogenous group of app servers a distributed system.)
  10. 13.

    Integration Components within the system must be well-integrated (often loosely

    coupled). Usually this implies real-time and bi- directional interfaces. (By this test, we wouldn’t leap to call a batch-integrated group of legacy backends a distributed system.)
  11. 15.

    I call these Complex-Linear Distributed Systems While there’s certainly more

    to them, linear scale is a common goal and a complex system is required to achieve it.
  12. 16.

    RECAP * distributed systems are quickly becoming the norm *

    most distributed systems are complex, fault tolerant, and scale linearly * henceforth I’ll use “distributed system” to mean a “complex-linear distributed system” --- why is this important to me, a full stack developer?
  13. 21.

    They’ve just been around the block a few times. In

    my experience, each lap of the block takes 2,000 to 3,000 hours. Developers who stick it out for 10-20k hours often reach the coveted ‘full-stack bad ass’ status. Full stack devs don’t always do *all* the work - they just could if they had to.
  14. 22.

    Full stack developers know: user experience user experience user experience

    business value business value business value user interface user interface user interface (some) design (some) design (some) design MVC Request Request Action Action API’s data access data access business logic business logic services services data modeling data modeling data modeling data formats data formats data formats kernel network network I/O I/O shell
  15. 23.

    Full stack developers can do: hiring fundraising mentoring pair programming

    user interface A/B testing user interface A/B testing test-driven API development test-driven API development database performance tuning database performance tuning linux kernel debugging linux kernel debugging
  16. 26.

    WHY? The definition of ‘full stack’ is the ability to

    build any level of the product or business. Businesses increasingly require distributed systems to be competitive. Full stack developers must become conversant and/or fluent in distributed systems.
  17. 28.
  18. 29.

    LOLJK Because of their experience, full stack developers can quickly

    acquire the tools and methodologies of distributed system building.
  19. 30.

    More skills: monitoring stacks DEVOPS fault tolerance distributed coordination partitioning

    serialization more about networking I/O and performance shell scripting
  20. 31.

    Distributed FS developers will need to learn: a) common software

    design heuristics for distributed systems: partitioning, serialization, fault tolerance, tradeoffs (like CAP), performance, capacity... path-oriented design: model the system as interdependent paths (e.g. read path / write path) and distribute complexity via priority
  21. 32.

    Often, performance is the essential ‘feature’ of the system/product. And

    thus, it becomes the key driver of the design. This is very different than traditional web/ app work where performance optimization can come later. Insufficient performance can undermine everything in the 11th hour #realtalk
  22. 33.

    Latency Numbers every distributed full stack developer should know execute

    CPU instruction 1 ns fetch from memory 100 ns send 2k over a 1Gbps network 20,000 ns read 1MB from memory 250,000 ns spinning disk seek 8,000,000 ns read 1MB from disk seq. 20,000,000 ns send packet US->UK & back 150,000,000 ns SSD disk seek 150,000 ns read 1MB seq. from SSD 1,000,000 ns round trip in same DC 500,000 ns run unit tests 60,000,000,000 ns github page load 161,000,000 ns hire a full stack engineer 2E+15 ns raise a round of funding 7.7E+15 ns From Norvig, others These numbers are guidelines and YMMV. You’ve been warned!
  23. 34.

    Distributed FS developers will also need to learn: b) a

    lot of “devops” provisioning; be it chef, puppet, or salt. oh, and vagrant (with vmware) monitoring; new relic is great, but it won’t be enough. embrace it. make friends with devops and ops folks. you will need their tools to have any idea of how your complex system is behaving.
  24. 35.

    You want one of these torx is a water-cooled hackintosh

    w/ 32GB RAM, SSD RAID, and 4960x1600 resolution
  25. 36.

    RECAP full stack developers have the smarts and panache to

    become DISTRIBUTED complex-linear full stack developers * learn new design heuristics * embrace devops
  26. 37.

    RECAP by embracing distributed system development, you can prevent many

    problems of scale AND open up new opportunities for feature and product development that wouldn’t be possible otherwise (lots of these have to do with data)
  27. 38.

    Practical ways to get down with distributed 1) Try out

    a distributed database. 2) Do a project with Storm or a similarly well-abstracted distributed system. 3) Integrate automated provisioning into your development workflow. 4) Chaos monkey ideally fault-tolerant parts of your existing system.
  28. 39.

    Resources Latency number gist https://gist.github.com/jboner/2841832 Teach yourself to program in

    10 years http://norvig.com/21-days.html What is a full stack developer? http://www.laurencegellert.com/2012/08/what-is-a-full- stack-developer/ The Full Stack, Part 1 https://www.facebook.com/note.php? note_id=461505383919