Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Culture

RyanFrantz
January 14, 2015

Continuous Culture

Culture is always evolving and doesn't come for free. This is the story of Etsy's journey from 4-person startup to an international organization and the culture that we all work to foster.

RyanFrantz

January 14, 2015
Tweet

More Decks by RyanFrantz

Other Decks in Business

Transcript

  1. @Ryan_Frantz #me • Ryan Frantz • Senior Operations Engineer at

    Etsy (Remote) • Built and managed teams in the past • Contributes to and helps maintain culture at Etsy
  2. @Ryan_Frantz By the Numbers (FY 2013) • Gross Marketplace Sales

    (GMS) $1.35 Billion • 44 million members, 1.2 million active sellers • 26 million active listings • Nearly Every Country Performing Transactions • > 600 Employees
  3. @Ryan_Frantz There WILL be a test… • Culture constantly develops

    • Communication is paramount • No divide between managers and engineers • Technology/product can attract people, but culture keeps them
  4. @Ryan_Frantz Core Engineering Principles • Just Ship • Every engineer

    can push to prod at any time • Favor simplicity and speed to ship • Maximize learning • ‘If it moves, graph it’
  5. @Ryan_Frantz Continuous Culture v1 • Mike Rembetsy & Patrick McDonnell

    • Presented at Velocity Conference • Etsy’s engineering culture evolution 2006-2012 • Slides: http://slidesha.re/1xYxZrG • Video: http://vimeo.com/51310058
  6. @Ryan_Frantz 2006 - 2008 Silos and Barriers • 4-person startup

    grows to employ 30 - 35 (including 15 ENG) • A very siloed culture, creates barriers to engineering collaboration • Bred initiatives like Sprouter - ‘Middleware of distrust!’ • Project dedicated to stopping engineers touching databases
  7. @Ryan_Frantz Management Changes • Maria Thomas from NPR promoted to

    CEO • Brings a clear understanding that community is very important • Prioritizes a culture that supports community • Chad Dickerson brought on as CTO • Brings a clearer focus to the engineering team • “This siloed culture cannot work, we need to start over”
  8. @Ryan_Frantz 2006-2008 Takeaways • Downtime was an accepted fact of

    life • It was even expected to a degree! • Engineering projects were often low impact • Community needs to be a technical focus • Survived the holiday season … just!
  9. @Ryan_Frantz 2009 Internal Improvements • As teams grow, big efforts

    in good communication • Daily standups • DevTools team • Deployinator • lighttpd -> Apache • Network solidified
  10. @Ryan_Frantz 2009 Takeaways • Built solid foundations • Invested in

    human capital • Strong collaboration • Infrastructure • A lot of reflection and finding an Engineering identity
  11. @Ryan_Frantz 2010 Standardization & Graphs • Moved to PHP &

    MySQL for everything • ‘If it moves, graph it’ • Ganglia, Graphite, FITB, Nagios, Naglite • Starting to use this data for work/life balance as well as technical/systems reasons
  12. @Ryan_Frantz 2010 Ideals • Blameless Postmortems • 1:1s as a

    core mgmt tool • Engineering career planning • Developer on-call • Use of A/B testing • Feature Flags & Ramp Up • Schema Change Thursday Management Ideals Engineering Ideals
  13. @Ryan_Frantz 2010 Takeaways • Reduce number of technologies used in

    development • Focus on technical visibility throughout the org • Developers responsible for code deploy (and on-call!) • Work/life balance focus
  14. @Ryan_Frantz 2011 Tech Highlights • End of long tail legacy

    silo holdovers (Sprouter gone!) • Non-standard technologies removed from production • Engineers receive 3 annual goals: • Speak at a conference • Write a blog post • Release open source software
  15. @Ryan_Frantz 2011 - Organizational Changes • Senior management to become

    more Engineering focused • Chad to CEO • Kellan promoted to CTO • Allspaw promoted to SVP of Operations • Consolidates importance of engineering culture to the very top of Etsy and increases stability
  16. @Ryan_Frantz 2011 Takeaways • Year of open sourced tools! •

    Statsd, Logster, Deployinator, Supergrep, Schemanator • Overall maturing of engineering - platform & people • Automation & config management solidified (Chef) • Security front and center • Security without negative impact to culture?
  17. @Ryan_Frantz 2011-2012 - A Focus on Security • Security alongside

    Dev & Ops as being integral to culture • Applying our core principles & learnings to security • Emphasis on security being a facilitator not a blocker • Security often enforced with terrible cultural impact • Build a human and effective security organization
  18. @Ryan_Frantz 2012 - Growth + Foster Our Values • Explosive

    growth in hiring, allow easy team transfers • Some major changes around product • Internationalization • High impact products (Shipping Labels, Gift Cards) • Became a certified B-Corp
  19. @Ryan_Frantz What’s a B-Corp ? • Aim to use the

    power of business to solve social & environmental issues • Impacts engineering in new and interesting ways: • Waste, Recycling, Compost, Flushes (Yes we graph them!) • Efficiency of our tech, data center usage & partners • ‘Make the world more like Etsy’ - Extending the culture
  20. @Ryan_Frantz 2012 - Technical Achievements • Create wholly separate Payments

    environment • Allows PCI compliance without disrupting the culture • Interface with the web stack via a restricted Internet facing API • Get serious on Data Science • Dedicated Hadoop cluster for full time data scientists • Taking some chances and broadening of our engineers
  21. @Ryan_Frantz 2012 Takeaways • Do what’s needed to sustain long

    term goals & not just keep the lights on • More headcount allows us to take chances • Focus on social/environmental impact, internally & externally with communities • Open source all of the things
  22. @Ryan_Frantz 2013 - An Interesting Year! • Had many of

    the hard engineering wins taken care of • Time to focus internally • No engineer can know everything any longer • Internal tooling • Designated Ops
  23. @Ryan_Frantz 2013 - Tooling • Morgue: Stores postmortem details (including

    IRC logs, images) • Opsweekly: Categorize and report Nagios alerts • Superbit: Allows simple querying of Vertica, Elasticsearch & big data by anyone who knows SQL • Catapult: Communicates experiments and related metrics (including conversion impact) • Begin a refocus on a Mobile/API First product vision
  24. @Ryan_Frantz 2013 - Designated Ops • Visibility of ongoing projects

    • Spreading knowledge • Advocating for engineers • Assignments rotate annually
  25. @Ryan_Frantz 2013 Takeaways • Democratization of data is made easier

    with tooling that levels access and allows interrogation by ALL • Designated Ops • Engineering invested in transparency & trust • The world doesn’t wait, mobile is the future
  26. @Ryan_Frantz 2014 - Organizational changes • Everyone pushes on their

    1st day • Yearly planning is restructured • San Francisco office opens • Acquire & integrate A Little Market
  27. @Ryan_Frantz Cultural Acquisition • Paris-based A Little Market (ALM) •

    Integrating another engineering culture can be tough • Language, timezone and human cultural differences • Can be very successful, but don’t underestimate
  28. @Ryan_Frantz 2014 - Technical • Migrate from Splunk to ELK

    (Elasticsearch/Logstash/Kibana) • Mobile First has increasing product focus • Mobile CI infrastructure embedded & ramped up • API First is huge effort and development push • Technical work for quality of life: On-call sleep tracking
  29. @Ryan_Frantz ELK Lessons • Changing a core tool requires huge

    communication investment • Understanding usage patterns • Accommodating that usage and factoring into the project • Describing the purpose of the project (not just about $$) • Technology can find its way into unexpected corners of your stack!
  30. @Ryan_Frantz Mobile First • Applying your principles and culture to

    the changing tech landscape is key • Continuous Deployment hard in the ‘App Store world’ • Continuous Integration still applies of course • Continuous Deployment becomes Continuous Delivery • Use API to enable feature flag-driven native apps
  31. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓

    ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  32. @Ryan_Frantz API First • Supporting the Mobile First push &

    diversity of clients • Adds security & agility • Embeds fundamental future resilience • Capacity planning becomes more challenging
  33. @Ryan_Frantz Why This Approach? • Continuous integration, Continuous Delivery •

    Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Improve Mean Time To Recovery • Stop stressing about releases!
  34. Single release Many releases 50K LOC/month Few opportunities for failure


    Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
  35. @Ryan_Frantz Sleep Tracking • Experiment with Jawbone UPs & Ops

    • Collect on-call sleep data • Analyze sleep lost when on-call • Focus on data for quality of life
  36. @Ryan_Frantz 2014 Takeaways • Another year of big growth +

    M&A • Integrating other engineering cultures inside your own is a challenge you should prepare for • Core tooling changes require great thought & communication • Mobile focus does not mean the end of always pushing • Tooling for happiness & W/L balance is a win for all
  37. @Ryan_Frantz 2015 Action Items • Culture is still king despite

    growth or M&A activity • It takes effort to keep it so however • Ensure our API is up to the job of supporting Mobile First • Ensure core tooling changes are understood & embraced by all • Communicate our engineering culture & history to new hires
  38. @Ryan_Frantz Conclusions • Culture doesn’t come for free, it takes

    continuous work • Iterate & improve - Even when you think you have ‘it’ • Don’t give in to potential disruptors like growth & security and let them destroy your culture
  39. @Ryan_Frantz Links / References Continuously Deploying Culture (Mike Rembetsy, Patrick

    McDonnell) Slides: http://slidesha.re/1xYxZrG Video: http://vimeo.com/51310058 Scaling Etsy, What Went Wrong, What Went Right (Ross Snyder) Slides & video: http://bit.ly/po8zIj Etsy’s Journey to Continuous Integration for Mobile Apps (Nassim Kammah) Blog post: http://bit.ly/1yiGWwc Mean Time to Sleep (Ryan Frantz, Laurie Denness) Slides, Blog post, code: 
 http://ryanfrantz.com/mtts/