Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Co-Workers As Customers: Lessons from Airbnb & Etsy

Co-Workers As Customers: Lessons from Airbnb & Etsy

Internal tools aren't known for their beautiful consumer UI, or extreme ease of use, but they are what keeps any company humming.
But this is changing. Airbnb and Etsy are among a handful of companies leading the way in developing not only simple, easy to use tools, but also ones that customers love - and by that I mean colleagues.
These tools make work more efficient and help everyone at the company deliver the best results. Learn from these industry leaders on how and why to build great internal tools and they'll share some of their prized (open source) possessions - Stats D, Deployinator, Chronos.

Daniel Schauenberg

March 10, 2014
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. CO-workers as customers
    Lessons from Airbnb & Etsy
    Harrison Shoff
    Daniel Schauenberg
    [email protected]
    [email protected]
    @hshoff
    @mrtazz

    View Slide

  2. Optional and somewhat awesome footnote text goes here.
    • StatsD
    • Deployinator
    • Morgue
    • chef-whitelist
    2
    Daniel
    Schauenberg
    Senior Software Engineer at Etsy’s
    Infrastructure Team

    View Slide

  3. New York City

    View Slide

  4. Portal Earrings
    by
    SVCharms

    View Slide

  5. Adventure Time
    Crochet Dolls
    by
    chichirevolver

    View Slide

  6. Etsy Stats
    6
    25M 18M 60M 1.5B
    members items listed monthly visitors page views per month

    View Slide

  7. Optional and somewhat awesome footnote text goes here.
    Etsy Stack
    • PHP/Ruby/Node.js/Python/Perl/Go
    • MySQL/Postgres/Vertica/Hadoop/Solr
    • MemcacheD/Redis/Gearman
    • StatsD/Logster/Graphite/Ganglia
    • Chef/Nagios/IRC
    7

    View Slide

  8. View Slide

  9. • Chronos
    • Photography Tool
    • JavaScript Style Guide
    9
    Harrison Shoff
    Design Engineer at Airbnb

    View Slide

  10. San Francisco

    View Slide

  11. Krakow, Poland
    11

    View Slide

  12. Treehouse - Vermont
    12

    View Slide

  13. 1 sq m House - Berlin, Germany
    13

    View Slide

  14. View Slide

  15. As of Dec 19, 2013
    Airbnb Stats
    15
    10M 550k 175 25
    6 Million in past year 250,000 past year
    nights booked properties countries languages

    View Slide

  16. Airbnb Stack
    • Ruby/Java/Node
    • Hadoop/Hive/Redshift/Mesos/Chronos
    • StatsD & Graphite (thanks Daniel)
    • MySQL/Redis/Dyson/S3/Memcached
    • Chef/Synapse/Nerve/Stemcell
    16

    View Slide

  17. Why build tools?

    View Slide

  18. Enable people

    View Slide

  19. Enable people
    1x 10x

    View Slide

  20. Reproducibility

    View Slide

  21. No dependency on providers

    View Slide

  22. Nothing says undocumented like,
    “run this command on this box”

    View Slide

  23. Not Invented Here Syndrome
    be careful about this

    View Slide

  24. Getting Started

    View Slide

  25. 3 warning signs that you’re
    missing a tool

    View Slide

  26. Outage or Incident

    View Slide

  27. It sucks because we have to do
    X then Y then Z
    X Y Z

    View Slide

  28. Simplify the process
    X
    Y
    Z

    View Slide

  29. “There should be a tool for this”

    View Slide

  30. 30
    Team Structure
    • 5 Engineers + 1 Manager
    • 1 Engineer per project
    • Consulting with developers, ops engineers
    • Most projects have at least 2 people who
    occasionally work on it
    • Each team is also building tools in their field
    • 12 engineers + 2 Managers
    • Data Infrastructure
    • 4 Engineers + 1 Manager
    • Ops Tools
    • 8 Engineers + 1 Manager
    • All building tools to support employees

    View Slide

  31. Open Source

    View Slide

  32. Why Open Source?
    Should we give away tools for free?
    32

    View Slide

  33. You probably owe
    most of
    your stack
    to Open Source

    View Slide

  34. Don't reinvent
    the wheel

    View Slide

  35. Item by WillowAndPoppy
    Give back to the community

    View Slide

  36. Item by XStitchMyHeart
    Contributions and Improvements by the community

    View Slide

  37. Open Source
    The good parts

    View Slide

  38. 38
    Good Examples
    Where it worked

    View Slide

  39. • Exist in almost any stack in some form
    • > 680 commits from > 120 contributors
    • Clients in almost any language
    • Only custom part is the config file
    • Pluggable backends
    • Flexible
    • Easy to get started
    • Switched over to the public version early
    • There is no internal repo
    39
    StatsD
    The Poster Child

    View Slide

  40. Morgue

    View Slide

  41. • Post Mortem Keeper
    • Timelines
    • IRC logs
    • Graphs
    • Images
    • Jira Tickets
    • Added feature API for Open Sourcing
    • Internal repo with Etsy features
    • Main version deployed from github.com
    41
    Morgue

    View Slide

  42. http://www.flickr.com/photos/tasselflower/95151097
    Kale
    (Skyline/Oculus)

    View Slide

  43. • Anomaly detection engine
    • Based on Graphite input data
    • Clearly defined interface/API
    • Started as a hack week project
    • Running the Open Source version from the start
    43
    Kale
    (Skyline/Oculus)

    View Slide

  44. Chronos
    JavaScript
    Style Guide
    How we write JavaScript

    View Slide

  45. View Slide

  46. • Mostly opinions
    • Helps with code reviews, on boarding
    • Big team, one codebase => consistency matters
    • Template for other developer teams to use
    • +1,400 forks
    • Only gets better over time
    46
    JavaScript
    Style Guide
    How we write JavaScript

    View Slide

  47. Chronos

    View Slide

  48. View Slide

  49. Optional and somewhat awesome footnote text goes here.
    • Information and stuff
    • So much information I can’t handle it
    • Look at all this data and stats and important findings
    • I can barely contain all this content!
    49
    • 10,000+ lines of Bash
    • 25 sleep statements
    • Single point of failure
    Broken Data
    Pipeline
    Before Chronos

    View Slide

  50. Optional and somewhat awesome footnote text goes here.
    • Information and stuff
    • So much information I can’t handle it
    • Look at all this data and stats and important findings
    • I can barely contain all this content!
    50
    • Mesos as a common platform for data infra (spark,
    shark, hadoop)
    • Express and visualize dependencies
    • Retries
    • < 2,000 lines of Scala
    • REST api
    • Web UI for analysts
    A Better Way
    Chronos

    View Slide

  51. 51

    View Slide

  52. View Slide

  53. Open Source
    The bad parts

    View Slide

  54. 54
    Bad Examples
    Where we wish we did things differently

    View Slide

  55. Deployinator
    55

    View Slide

  56. • Does all the deploys
    • One button deploys
    • Makes it easy to add new stacks
    • Open Sourced during a conference in 2010
    • We wanted to get it out there
    • Had to rip too much stuff out to make it work
    internally
    • Never took the time to get back in sync
    56
    Deployinator
    The Rushed One

    View Slide

  57. Dashboards
    57

    View Slide

  58. • Powers all Etsy dashboards
    • Makes it extremely easy to add new ones
    • Supports Graphite and Ganglia
    • Heavily tied to Etsy web code
    • No clear abstraction
    • Open sourced a snapshot
    • Periodic updates
    58
    Dashboards
    The Deep Integration

    View Slide

  59. View Slide

  60. • Parses loglines and emits metrics
    • Support for Graphite and Ganglia
    • Individual parsers are just Python classes
    • Open sourced without switching over to the public
    version
    • Popular project so it diverged quickly
    • Internal deploy not set up to sync easily
    • On-going effort to run the Open Source version
    60
    Logster
    The missed opportunity

    View Slide

  61. 61

    View Slide

  62. 62
    +750 jobs

    View Slide

  63. 63
    +750 jobs
    We only tested the UI with 300 jobs…

    View Slide

  64. View Slide

  65. Internal Fork

    View Slide

  66. Internal Fork
    Supporting 2 projects
    Maintenance priority

    View Slide

  67. View Slide

  68. Back on track

    View Slide

  69. Difficulties
    Why open source?

    View Slide

  70. http://www.flickr.com/photos/allthosedetails/7665289260
    Extra
    Work

    View Slide

  71. View Slide

  72. The fork is always
    the easier route

    View Slide

  73. You might be solving
    a very specific problem

    View Slide

  74. People, deal with it!

    View Slide

  75. • Enforces some good constraints about testability and
    separate of concerns
    • Great for recruiting
    • New hires are familiar with your tools
    • Freenode/GitHub does a lot of work for you
    75
    Experiences
    Why Open Source?

    View Slide

  76. • Have a Freenode IRC channel (#codeascraft)
    • Have an active GitHub presence
    • Think about Open Source right from the start
    • Do not ever run an internal fork
    • Seriously don’t
    76
    Tips
    Why Open Source?

    View Slide

  77. • All about enablement
    • Decrease the time and frustration spent waiting
    • Engineer happiness++
    • Fast feedback (continuous integration, continuous
    deployment)
    77
    Tooling Culture
    Why is a tooling culture important?

    View Slide

  78. • Bootcamp with Tools teams
    • Tools are for the people, not the individual
    • Let others contribute and extend them
    • Have an open ear/IRC channel
    • Make it easy to create new tools
    78
    Tips
    How to build a tooling culture?

    View Slide

  79. Build
    Internal
    Tools
    http://www.flickr.com/photos/zzpza/3269784239

    View Slide

  80. Enable
    a tooling
    culture
    http://www.flickr.com/photos/usacehq/4920584963

    View Slide

  81. Open source
    needs
    maintenance
    Item by LovelyLittleTrove

    View Slide

  82. Open source
    is great
    Item by SteelPetalPress

    View Slide

  83. Thanks!
    Questions?
    Harrison Shoff
    Daniel Schauenberg
    [email protected]
    [email protected]
    @hshoff
    @mrtazz

    View Slide