Co-Workers As Customers: Lessons from Airbnb & Etsy

Internal tools aren't known for their beautiful consumer UI, or extreme ease of use, but they are what keeps any company humming.
But this is changing. Airbnb and Etsy are among a handful of companies leading the way in developing not only simple, easy to use tools, but also ones that customers love - and by that I mean colleagues.
These tools make work more efficient and help everyone at the company deliver the best results. Learn from these industry leaders on how and why to build great internal tools and they'll share some of their prized (open source) possessions - Stats D, Deployinator, Chronos.

Transcript

1. CO-workers as customers
   Lessons from Airbnb & Etsy
   Harrison Shoff
   Daniel Schauenberg
   harry@airbnb.com
   dschauenberg@etsy.com
   @hshoff
   @mrtazz
   

   View Slide

2. Optional and somewhat awesome footnote text goes here.
   • StatsD
   • Deployinator
   • Morgue
   • chef-whitelist
   2
   Daniel
   Schauenberg
   Senior Software Engineer at Etsy’s
   Infrastructure Team
   

   View Slide

3. New York City
   

   View Slide

4. Portal Earrings
   by
   SVCharms
   

   View Slide

5. Adventure Time
   Crochet Dolls
   by
   chichirevolver
   

   View Slide

6. Etsy Stats
   6
   25M 18M 60M 1.5B
   members items listed monthly visitors page views per month
   

   View Slide

7. Optional and somewhat awesome footnote text goes here.
   Etsy Stack
   • PHP/Ruby/Node.js/Python/Perl/Go
   • MySQL/Postgres/Vertica/Hadoop/Solr
   • MemcacheD/Redis/Gearman
   • StatsD/Logster/Graphite/Ganglia
   • Chef/Nagios/IRC
   7
   

   View Slide

8. View Slide

9. • Chronos
   • Photography Tool
   • JavaScript Style Guide
   9
   Harrison Shoff
   Design Engineer at Airbnb
   

   View Slide

10. San Francisco
    

    View Slide

11. Krakow, Poland
    11
    

    View Slide

12. Treehouse - Vermont
    12
    

    View Slide

13. 1 sq m House - Berlin, Germany
    13
    

    View Slide

14. View Slide

15. As of Dec 19, 2013
    Airbnb Stats
    15
    10M 550k 175 25
    6 Million in past year 250,000 past year
    nights booked properties countries languages
    

    View Slide

16. Airbnb Stack
    • Ruby/Java/Node
    • Hadoop/Hive/Redshift/Mesos/Chronos
    • StatsD & Graphite (thanks Daniel)
    • MySQL/Redis/Dyson/S3/Memcached
    • Chef/Synapse/Nerve/Stemcell
    16
    

    View Slide

17. Why build tools?
    

    View Slide

18. Enable people
    

    View Slide

19. Enable people
    1x 10x
    

    View Slide

20. Reproducibility
    

    View Slide

21. No dependency on providers
    

    View Slide

22. Nothing says undocumented like,
    “run this command on this box”
    

    View Slide

23. Not Invented Here Syndrome
    be careful about this
    

    View Slide

24. Getting Started
    

    View Slide

25. 3 warning signs that you’re
    missing a tool
    

    View Slide

26. Outage or Incident
    

    View Slide

27. It sucks because we have to do
    X then Y then Z
    X Y Z
    

    View Slide

28. Simplify the process
    X
    Y
    Z
    

    View Slide

29. “There should be a tool for this”
    

    View Slide

30. 30
    Team Structure
    • 5 Engineers + 1 Manager
    • 1 Engineer per project
    • Consulting with developers, ops engineers
    • Most projects have at least 2 people who
    occasionally work on it
    • Each team is also building tools in their field
    • 12 engineers + 2 Managers
    • Data Infrastructure
    • 4 Engineers + 1 Manager
    • Ops Tools
    • 8 Engineers + 1 Manager
    • All building tools to support employees
    

    View Slide

31. Open Source
    

    View Slide

32. Why Open Source?
    Should we give away tools for free?
    32
    

    View Slide

33. You probably owe
    most of
    your stack
    to Open Source
    

    View Slide

34. Don't reinvent
    the wheel
    

    View Slide

35. Item by WillowAndPoppy
    Give back to the community
    

    View Slide

36. Item by XStitchMyHeart
    Contributions and Improvements by the community
    

    View Slide

37. Open Source
    The good parts
    

    View Slide

38. 38
    Good Examples
    Where it worked
    

    View Slide

39. • Exist in almost any stack in some form
    • > 680 commits from > 120 contributors
    • Clients in almost any language
    • Only custom part is the config file
    • Pluggable backends
    • Flexible
    • Easy to get started
    • Switched over to the public version early
    • There is no internal repo
    39
    StatsD
    The Poster Child
    

    View Slide

40. Morgue
    

    View Slide

41. • Post Mortem Keeper
    • Timelines
    • IRC logs
    • Graphs
    • Images
    • Jira Tickets
    • Added feature API for Open Sourcing
    • Internal repo with Etsy features
    • Main version deployed from github.com
    41
    Morgue
    

    View Slide

42. http://www.flickr.com/photos/tasselflower/95151097
    Kale
    (Skyline/Oculus)
    

    View Slide

43. • Anomaly detection engine
    • Based on Graphite input data
    • Clearly defined interface/API
    • Started as a hack week project
    • Running the Open Source version from the start
    43
    Kale
    (Skyline/Oculus)
    

    View Slide

44. Chronos
    JavaScript
    Style Guide
    How we write JavaScript
    

    View Slide

45. View Slide

46. • Mostly opinions
    • Helps with code reviews, on boarding
    • Big team, one codebase => consistency matters
    • Template for other developer teams to use
    • +1,400 forks
    • Only gets better over time
    46
    JavaScript
    Style Guide
    How we write JavaScript
    

    View Slide

47. Chronos
    

    View Slide

48. View Slide

49. Optional and somewhat awesome footnote text goes here.
    • Information and stuff
    • So much information I can’t handle it
    • Look at all this data and stats and important findings
    • I can barely contain all this content!
    49
    • 10,000+ lines of Bash
    • 25 sleep statements
    • Single point of failure
    Broken Data
    Pipeline
    Before Chronos
    

    View Slide

50. Optional and somewhat awesome footnote text goes here.
    • Information and stuff
    • So much information I can’t handle it
    • Look at all this data and stats and important findings
    • I can barely contain all this content!
    50
    • Mesos as a common platform for data infra (spark,
    shark, hadoop)
    • Express and visualize dependencies
    • Retries
    • < 2,000 lines of Scala
    • REST api
    • Web UI for analysts
    A Better Way
    Chronos
    

    View Slide

51. View Slide

52. View Slide

53. Open Source
    The bad parts
    

    View Slide

54. 54
    Bad Examples
    Where we wish we did things differently
    

    View Slide

55. Deployinator
    55
    

    View Slide

56. • Does all the deploys
    • One button deploys
    • Makes it easy to add new stacks
    • Open Sourced during a conference in 2010
    • We wanted to get it out there
    • Had to rip too much stuff out to make it work
    internally
    • Never took the time to get back in sync
    56
    Deployinator
    The Rushed One
    

    View Slide

57. Dashboards
    57
    

    View Slide

58. • Powers all Etsy dashboards
    • Makes it extremely easy to add new ones
    • Supports Graphite and Ganglia
    • Heavily tied to Etsy web code
    • No clear abstraction
    • Open sourced a snapshot
    • Periodic updates
    58
    Dashboards
    The Deep Integration
    

    View Slide

59. View Slide

60. • Parses loglines and emits metrics
    • Support for Graphite and Ganglia
    • Individual parsers are just Python classes
    • Open sourced without switching over to the public
    version
    • Popular project so it diverged quickly
    • Internal deploy not set up to sync easily
    • On-going effort to run the Open Source version
    60
    Logster
    The missed opportunity
    

    View Slide

61. View Slide

62. 62
    +750 jobs
    

    View Slide

63. 63
    +750 jobs
    We only tested the UI with 300 jobs…
    

    View Slide

64. View Slide

65. Internal Fork
    

    View Slide

66. Internal Fork
    Supporting 2 projects
    Maintenance priority
    

    View Slide

67. View Slide

68. Back on track
    

    View Slide

69. Difficulties
    Why open source?
    

    View Slide

70. http://www.flickr.com/photos/allthosedetails/7665289260
    Extra
    Work
    

    View Slide

71. View Slide

72. The fork is always
    the easier route
    

    View Slide

73. You might be solving
    a very specific problem
    

    View Slide

74. People, deal with it!
    

    View Slide

75. • Enforces some good constraints about testability and
    separate of concerns
    • Great for recruiting
    • New hires are familiar with your tools
    • Freenode/GitHub does a lot of work for you
    75
    Experiences
    Why Open Source?
    

    View Slide

76. • Have a Freenode IRC channel (#codeascraft)
    • Have an active GitHub presence
    • Think about Open Source right from the start
    • Do not ever run an internal fork
    • Seriously don’t
    76
    Tips
    Why Open Source?
    

    View Slide

77. • All about enablement
    • Decrease the time and frustration spent waiting
    • Engineer happiness++
    • Fast feedback (continuous integration, continuous
    deployment)
    77
    Tooling Culture
    Why is a tooling culture important?
    

    View Slide

78. • Bootcamp with Tools teams
    • Tools are for the people, not the individual
    • Let others contribute and extend them
    • Have an open ear/IRC channel
    • Make it easy to create new tools
    78
    Tips
    How to build a tooling culture?
    

    View Slide

79. Build
    Internal
    Tools
    http://www.flickr.com/photos/zzpza/3269784239
    

    View Slide

80. Enable
    a tooling
    culture
    http://www.flickr.com/photos/usacehq/4920584963
    

    View Slide

81. Open source
    needs
    maintenance
    Item by LovelyLittleTrove
    

    View Slide

82. Open source
    is great
    Item by SteelPetalPress
    

    View Slide

83. Thanks!
    Questions?
    Harrison Shoff
    Daniel Schauenberg
    harry@airbnb.com
    dschauenberg@etsy.com
    @hshoff
    @mrtazz
    

    View Slide