Co-Workers As Customers: Lessons from Airbnb & Etsy

Co-Workers As Customers: Lessons from Airbnb & Etsy

Internal tools aren't known for their beautiful consumer UI, or extreme ease of use, but they are what keeps any company humming.
But this is changing. Airbnb and Etsy are among a handful of companies leading the way in developing not only simple, easy to use tools, but also ones that customers love - and by that I mean colleagues.
These tools make work more efficient and help everyone at the company deliver the best results. Learn from these industry leaders on how and why to build great internal tools and they'll share some of their prized (open source) possessions - Stats D, Deployinator, Chronos.

89e0ad1229121f46047977ac547bd7b4?s=128

Daniel Schauenberg

March 10, 2014
Tweet

Transcript

  1. CO-workers as customers Lessons from Airbnb & Etsy Harrison Shoff

    Daniel Schauenberg harry@airbnb.com dschauenberg@etsy.com @hshoff @mrtazz
  2. Optional and somewhat awesome footnote text goes here. • StatsD

    • Deployinator • Morgue • chef-whitelist 2 Daniel Schauenberg Senior Software Engineer at Etsy’s Infrastructure Team
  3. New York City

  4. Portal Earrings by SVCharms

  5. Adventure Time Crochet Dolls by chichirevolver

  6. Etsy Stats 6 25M 18M 60M 1.5B members items listed

    monthly visitors page views per month
  7. Optional and somewhat awesome footnote text goes here. Etsy Stack

    • PHP/Ruby/Node.js/Python/Perl/Go • MySQL/Postgres/Vertica/Hadoop/Solr • MemcacheD/Redis/Gearman • StatsD/Logster/Graphite/Ganglia • Chef/Nagios/IRC 7
  8. None
  9. • Chronos • Photography Tool • JavaScript Style Guide 9

    Harrison Shoff Design Engineer at Airbnb
  10. San Francisco

  11. Krakow, Poland 11

  12. Treehouse - Vermont 12

  13. 1 sq m House - Berlin, Germany 13

  14. None
  15. As of Dec 19, 2013 Airbnb Stats 15 10M 550k

    175 25 6 Million in past year 250,000 past year nights booked properties countries languages
  16. Airbnb Stack • Ruby/Java/Node • Hadoop/Hive/Redshift/Mesos/Chronos • StatsD & Graphite

    (thanks Daniel) • MySQL/Redis/Dyson/S3/Memcached • Chef/Synapse/Nerve/Stemcell 16
  17. Why build tools?

  18. Enable people

  19. Enable people 1x 10x

  20. Reproducibility

  21. No dependency on providers

  22. Nothing says undocumented like, “run this command on this box”

  23. Not Invented Here Syndrome be careful about this

  24. Getting Started

  25. 3 warning signs that you’re missing a tool

  26. Outage or Incident

  27. It sucks because we have to do X then Y

    then Z X Y Z
  28. Simplify the process X Y Z

  29. “There should be a tool for this”

  30. 30 Team Structure • 5 Engineers + 1 Manager •

    1 Engineer per project • Consulting with developers, ops engineers • Most projects have at least 2 people who occasionally work on it • Each team is also building tools in their field • 12 engineers + 2 Managers • Data Infrastructure • 4 Engineers + 1 Manager • Ops Tools • 8 Engineers + 1 Manager • All building tools to support employees
  31. Open Source

  32. Why Open Source? Should we give away tools for free?

    32
  33. You probably owe most of your stack to Open Source

  34. Don't reinvent the wheel

  35. Item by WillowAndPoppy Give back to the community

  36. Item by XStitchMyHeart Contributions and Improvements by the community

  37. Open Source The good parts

  38. 38 Good Examples Where it worked

  39. • Exist in almost any stack in some form •

    > 680 commits from > 120 contributors • Clients in almost any language • Only custom part is the config file • Pluggable backends • Flexible • Easy to get started • Switched over to the public version early • There is no internal repo 39 StatsD The Poster Child
  40. Morgue

  41. • Post Mortem Keeper • Timelines • IRC logs •

    Graphs • Images • Jira Tickets • Added feature API for Open Sourcing • Internal repo with Etsy features • Main version deployed from github.com 41 Morgue
  42. http://www.flickr.com/photos/tasselflower/95151097 Kale (Skyline/Oculus)

  43. • Anomaly detection engine • Based on Graphite input data

    • Clearly defined interface/API • Started as a hack week project • Running the Open Source version from the start 43 Kale (Skyline/Oculus)
  44. Chronos JavaScript Style Guide How we write JavaScript

  45. None
  46. • Mostly opinions • Helps with code reviews, on boarding

    • Big team, one codebase => consistency matters • Template for other developer teams to use • +1,400 forks • Only gets better over time 46 JavaScript Style Guide How we write JavaScript
  47. Chronos

  48. None
  49. Optional and somewhat awesome footnote text goes here. • Information

    and stuff • So much information I can’t handle it • Look at all this data and stats and important findings • I can barely contain all this content! 49 • 10,000+ lines of Bash • 25 sleep statements • Single point of failure Broken Data Pipeline Before Chronos
  50. Optional and somewhat awesome footnote text goes here. • Information

    and stuff • So much information I can’t handle it • Look at all this data and stats and important findings • I can barely contain all this content! 50 • Mesos as a common platform for data infra (spark, shark, hadoop) • Express and visualize dependencies • Retries • < 2,000 lines of Scala • REST api • Web UI for analysts A Better Way Chronos
  51. 51

  52. None
  53. Open Source The bad parts

  54. 54 Bad Examples Where we wish we did things differently

  55. Deployinator 55

  56. • Does all the deploys • One button deploys •

    Makes it easy to add new stacks • Open Sourced during a conference in 2010 • We wanted to get it out there • Had to rip too much stuff out to make it work internally • Never took the time to get back in sync 56 Deployinator The Rushed One
  57. Dashboards 57

  58. • Powers all Etsy dashboards • Makes it extremely easy

    to add new ones • Supports Graphite and Ganglia • Heavily tied to Etsy web code • No clear abstraction • Open sourced a snapshot • Periodic updates 58 Dashboards The Deep Integration
  59. None
  60. • Parses loglines and emits metrics • Support for Graphite

    and Ganglia • Individual parsers are just Python classes • Open sourced without switching over to the public version • Popular project so it diverged quickly • Internal deploy not set up to sync easily • On-going effort to run the Open Source version 60 Logster The missed opportunity
  61. 61

  62. 62 +750 jobs

  63. 63 +750 jobs We only tested the UI with 300

    jobs…
  64. None
  65. Internal Fork

  66. Internal Fork Supporting 2 projects Maintenance priority

  67. None
  68. Back on track

  69. Difficulties Why open source?

  70. http://www.flickr.com/photos/allthosedetails/7665289260 Extra Work

  71. None
  72. The fork is always the easier route

  73. You might be solving a very specific problem

  74. People, deal with it!

  75. • Enforces some good constraints about testability and separate of

    concerns • Great for recruiting • New hires are familiar with your tools • Freenode/GitHub does a lot of work for you 75 Experiences Why Open Source?
  76. • Have a Freenode IRC channel (#codeascraft) • Have an

    active GitHub presence • Think about Open Source right from the start • Do not ever run an internal fork • Seriously don’t 76 Tips Why Open Source?
  77. • All about enablement • Decrease the time and frustration

    spent waiting • Engineer happiness++ • Fast feedback (continuous integration, continuous deployment) 77 Tooling Culture Why is a tooling culture important?
  78. • Bootcamp with Tools teams • Tools are for the

    people, not the individual • Let others contribute and extend them • Have an open ear/IRC channel • Make it easy to create new tools 78 Tips How to build a tooling culture?
  79. Build Internal Tools http://www.flickr.com/photos/zzpza/3269784239

  80. Enable a tooling culture http://www.flickr.com/photos/usacehq/4920584963

  81. Open source needs maintenance Item by LovelyLittleTrove

  82. Open source is great Item by SteelPetalPress

  83. Thanks! Questions? Harrison Shoff Daniel Schauenberg harry@airbnb.com dschauenberg@etsy.com @hshoff @mrtazz