Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Co-Workers As Customers: Lessons from Airbnb & Etsy

Co-Workers As Customers: Lessons from Airbnb & Etsy

Internal tools aren't known for their beautiful consumer UI, or extreme ease of use, but they are what keeps any company humming.
But this is changing. Airbnb and Etsy are among a handful of companies leading the way in developing not only simple, easy to use tools, but also ones that customers love - and by that I mean colleagues.
These tools make work more efficient and help everyone at the company deliver the best results. Learn from these industry leaders on how and why to build great internal tools and they'll share some of their prized (open source) possessions - Stats D, Deployinator, Chronos.

Daniel Schauenberg

March 10, 2014

More Decks by Daniel Schauenberg

Other Decks in Technology


  1. Optional and somewhat awesome footnote text goes here. • StatsD

    • Deployinator • Morgue • chef-whitelist 2 Daniel Schauenberg Senior Software Engineer at Etsy’s Infrastructure Team
  2. Etsy Stats 6 25M 18M 60M 1.5B members items listed

    monthly visitors page views per month
  3. Optional and somewhat awesome footnote text goes here. Etsy Stack

    • PHP/Ruby/Node.js/Python/Perl/Go • MySQL/Postgres/Vertica/Hadoop/Solr • MemcacheD/Redis/Gearman • StatsD/Logster/Graphite/Ganglia • Chef/Nagios/IRC 7
  4. • Chronos • Photography Tool • JavaScript Style Guide 9

    Harrison Shoff Design Engineer at Airbnb
  5. As of Dec 19, 2013 Airbnb Stats 15 10M 550k

    175 25 6 Million in past year 250,000 past year nights booked properties countries languages
  6. Airbnb Stack • Ruby/Java/Node • Hadoop/Hive/Redshift/Mesos/Chronos • StatsD & Graphite

    (thanks Daniel) • MySQL/Redis/Dyson/S3/Memcached • Chef/Synapse/Nerve/Stemcell 16
  7. 30 Team Structure • 5 Engineers + 1 Manager •

    1 Engineer per project • Consulting with developers, ops engineers • Most projects have at least 2 people who occasionally work on it • Each team is also building tools in their field • 12 engineers + 2 Managers • Data Infrastructure • 4 Engineers + 1 Manager • Ops Tools • 8 Engineers + 1 Manager • All building tools to support employees
  8. • Exist in almost any stack in some form •

    > 680 commits from > 120 contributors • Clients in almost any language • Only custom part is the config file • Pluggable backends • Flexible • Easy to get started • Switched over to the public version early • There is no internal repo 39 StatsD The Poster Child
  9. • Post Mortem Keeper • Timelines • IRC logs •

    Graphs • Images • Jira Tickets • Added feature API for Open Sourcing • Internal repo with Etsy features • Main version deployed from github.com 41 Morgue
  10. • Anomaly detection engine • Based on Graphite input data

    • Clearly defined interface/API • Started as a hack week project • Running the Open Source version from the start 43 Kale (Skyline/Oculus)
  11. • Mostly opinions • Helps with code reviews, on boarding

    • Big team, one codebase => consistency matters • Template for other developer teams to use • +1,400 forks • Only gets better over time 46 JavaScript Style Guide How we write JavaScript
  12. Optional and somewhat awesome footnote text goes here. • Information

    and stuff • So much information I can’t handle it • Look at all this data and stats and important findings • I can barely contain all this content! 49 • 10,000+ lines of Bash • 25 sleep statements • Single point of failure Broken Data Pipeline Before Chronos
  13. Optional and somewhat awesome footnote text goes here. • Information

    and stuff • So much information I can’t handle it • Look at all this data and stats and important findings • I can barely contain all this content! 50 • Mesos as a common platform for data infra (spark, shark, hadoop) • Express and visualize dependencies • Retries • < 2,000 lines of Scala • REST api • Web UI for analysts A Better Way Chronos
  14. 51

  15. • Does all the deploys • One button deploys •

    Makes it easy to add new stacks • Open Sourced during a conference in 2010 • We wanted to get it out there • Had to rip too much stuff out to make it work internally • Never took the time to get back in sync 56 Deployinator The Rushed One
  16. • Powers all Etsy dashboards • Makes it extremely easy

    to add new ones • Supports Graphite and Ganglia • Heavily tied to Etsy web code • No clear abstraction • Open sourced a snapshot • Periodic updates 58 Dashboards The Deep Integration
  17. • Parses loglines and emits metrics • Support for Graphite

    and Ganglia • Individual parsers are just Python classes • Open sourced without switching over to the public version • Popular project so it diverged quickly • Internal deploy not set up to sync easily • On-going effort to run the Open Source version 60 Logster The missed opportunity
  18. 61

  19. • Enforces some good constraints about testability and separate of

    concerns • Great for recruiting • New hires are familiar with your tools • Freenode/GitHub does a lot of work for you 75 Experiences Why Open Source?
  20. • Have a Freenode IRC channel (#codeascraft) • Have an

    active GitHub presence • Think about Open Source right from the start • Do not ever run an internal fork • Seriously don’t 76 Tips Why Open Source?
  21. • All about enablement • Decrease the time and frustration

    spent waiting • Engineer happiness++ • Fast feedback (continuous integration, continuous deployment) 77 Tooling Culture Why is a tooling culture important?
  22. • Bootcamp with Tools teams • Tools are for the

    people, not the individual • Let others contribute and extend them • Have an open ear/IRC channel • Make it easy to create new tools 78 Tips How to build a tooling culture?